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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  Computer-Based  Educational  Technology  Team  of  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  performs 
research  and  development  in  eares  of  educational  technology  that  apply  to 
military  training.  Of  interest  are  computer-based  instructional  delivery 
systems  that  focus  on  developing  the  accompanying  instructional  course¬ 
ware  in  the  most  efficient  and  cost-effective  manner  possible.  Develop¬ 
ment  and  implementation  of  such  systems  will  help  solve  the  problem  of 
training  individuals  to  produce  good  courseware  in  a  reasonable  time,  at 
an  acceptable  cost. 

This  Technical  Report  describes  a  development  and  feasibility  demon¬ 
stration  of  two  author  aids  designed  to  assist  individuals  in  developing 
tests  and  instruction.  The  project  was  funded  jointly  by  ARI  and  the 
Defense  Advanced  Research  Projects  Agency  (DARPA) .  To  accomplish  this 
research,  ARI's  resources  were  augmented  by  contract  DAHC  19-76-C-0041  with 
the  Human  Resources  Research  Organization,  an  organization  selected  as  hav¬ 
ing  unique  capabilities  for  research  and  development  in  this  area. 

Personnel  at  the  U.S.  Army  Engineer  School  (USAES) ,  Fort  Belvoir,  Va., 
provided  guidance  and  assistance  throughout  the  project:  Dr.  Everett 
Rompf,  Mr.  Jack  Ainsworth,  LTC  Ernest  Larson,  MAJ  John  Harvey,  MAJ  Ramile  R. 
Rebello,  1LT  D.  Bunn,  SFC  Alton  J.  Blanchard,  and  SFC  Leon  M.  Loomis.  In 
addition.  Dr.  James  Kraatz,  PLATO  Services  Organization,  Computer-Based 
Educational  Research  Laboratory,  University  of  Illinois  at  Urbana,  and 
Ms.  Beverly  Hunter  and  Mr.  Richard  Rosenblatt  of  HumRRO  also  contributed 
to  the  research  effort. 

The  entire  research  work  unit  area  is  responsive  to  the  requirements 
of  TDT&E  Project  2Q762717A764,  "Educational  and  Training  Technology,"  the 
1977  ARI  Work  Program. 
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ON-LINE  AUTHORING  AIDS  FOR  INSTRUCTIONAL  DESIGN 


BRIEF 


Requirement: 

The  purpose  was  to  examine  the  feasibility  of  providing  "how  to  do 
it"  guidance  (authoring  aids)  for  the  instructional  design  and  development 
tasks  identified  by  the  Interservice  Procedures  for  Instructional  Systems 
Development  (IPISD)  model.  The  usefulness  of  the  IPISD  model  depends  on 
authoring  aids  which  enable  training  personnel  to  translate  IPISD  pro¬ 
cedures  into  instructional  products.  The  authoring  aids  developed  by  this 
research  should  be  useful  for  computer-based  and  off-line  instruction  and 
be  general izable  to  differing  subject  matter  areas. 


Procedure: 

Authoring  aids  were  constructed,  implemented,  and  tested.  The  author¬ 
ing  aids  were  developed  to  be  used  on  the  PLATO  IV  Computer-Assisted  In¬ 
struction  (CAI)  system.  The  first  step  produced  flowcharts  which  detailed 
the  steps  of  the  IPISD  Blocks  II. 2  (Develop  Tests)  and  III. 4  (Develop  In¬ 
struction)  .  On-line  author  aids  as  well  as  off-line  versions  were  produced 
to  assist  the  author  in  preparing  materials  for  CAI  and  non-CAI  delivery 
of  instruction. 

Three  levels  of  evaluation  were  conducted.  An  informal  evaluation 
on  existing  IPISD  materials  was  performed,  and  a  formative  evaluation  on 
the  newly  developed  authoring  aids.  Finally,  the  instructional  materials 
were  evaluated  by  military  authors  and  administered  to  U.S.  Army  Engineer 
School  trainees. 


Findings : 

The  feasibility  of  on-line  aids  for  implementing  IPISD  Blocks  II. 2 
(Develop  Tests)  and  III. 4  (Develop  Instruction)  was  demonstrated  through 
the  evaluations.  User  acceptance  of  the  aids  was  hiqh,  and  the  time  re¬ 
quired  for  development  of  test  and  lesson  material  has  been  significantly 
reduced . 


Utilization  of  Findings: 

Based  on  these  findings,  the  development  of  authoring  aids  for  addi¬ 
tional  blocks  of  the  IPISD  model  was  initiated. 
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Chapter  I 
BACKGROUND 


The  purpose  of  this  chapter  is  to  provide  an  overview  of  the  state- 
of-the-art  related  to  instructional  systems  design  and  evaluation.  Major 
subjects  addressed  in  this  chapter  are: 

•  Systems  Approach  to  Training 

•  Problems  of  Implementing  Instructional  System  Development  Models 

•  The  Need  for  Author  Aids 

•  Approaches  and  Techniques  for  Evaluating  Instruction 

SYSTEMS  APPROACH  TO  TRAINING 

A  revolution  in  the  technology  of  training  within  the  military  and 
industry  began  when  the  systems  concept  was  applied  to  the  development  and 
conduct  of  training.  Even  now,  afti  '  25  years,  the  full  potential  of 
applying  the  systems  approach  to  improve  the  effectiveness  of  instruction, 
improve  on-the-job  performance  and  lower  the  cost  of  training  has  not  been 
realized.  Even  so,  results  from  applications  of  the  systems  approach  in 
terms  of  improved  instruction,  increased  relevance  in  what  is  taught,  and 
lowered  costs  have  been  so  dramatic  that,  at  present,  the  systems  approach 
to  training  has  permeated  civilian  training  (and  education  as  well) — 
training  in  business  and  industry,  training  in  the  military  services,  and 
training  in  other  agencies  of  the  federal  government. 

Many  different  names,  terms,  and  variations  are  or  have  been  used  for 
the  systems  approach  to  training.  Some  of  the  names  are:  "systems 
engineering  of  training,"  "curriculum  engineering,"  "systems  approach  to 
training,"  "instructional  systems  development,"  "training  situation  analysis," 
"modern  instructional  technology."  Even  among  those  using  a  particular 
name,  there  are  many  divergences  in  definitions,  particulars  of  technique 
and  procedures,  and  effectiveness  with  which  the  systems  approach  is 
applied.  Despite  such  variations,  the  cormon  thrust  and  orientation  of 
these  applications  is  pre-eminent,  specially  as  they  contrast  with  tradi¬ 
tional  approaches  to  training. 

The  essence  of  the  systems  approach  to  training  rests  in  identifying 
explicit  end  states  that  are  to  be  achieved  through  training  and  in  defining 
sets  of  orderly,  objective,  and  explicit  procedures  to  do  that  which  is 
necessary  to  achieve  these  end  states  in  the  most  comprehensive,  reliable, 
effective  and  efficient  manner. 

The  systems  approach  defines  a  process  which  foom'cs  upon  for 
that  is  ultimately  to  be  performed  and  upon  tho  inJivuiu*',!  who  is  to  learn 
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to  perform  that  Job.  Traditional  approaches  bv  contrast  „ 

tr«Sde«  WU  t?ZZ  f«"?Uy>  °"ly  *«•"”“■*“**  pertinent  to  „„at 

In  addition,  most  traditional  approaches  place  the  burden  for  In  for 

-PO"  'fe  etudents  rather  than  on  the  iLt™So»e[ 
materials.  Whether  the  instruction  is  rapid  or  slow  comnlex  i  , , 
t^^iudent^ustada£t.  By  contrast,  in  the  systems ’approach  it  is  ’ 
feasible  to  engineer  flexibility  into  the  instruction  and  so  to  adaot 

the  instt^^a^jjst^  to  individual  differences  among  the  students 
Special  consideration  is  given  to:  K  students. 

•  Evaluation  of  the  needs  of  each  individual  student. 

#  "ature  of  instructional  content  to  be  imparted. 

i instructional,  decision  rules  that  mediate  between  student 
needs  and  instructional  content. 

The  systems  approach  is  just  what  the  name  implies:  a  systematic 

£lliebe  t°r  hr^1?8  the  deSlred  Products  of  training  and  selecting  what 
e  taught,  how  it  will  be  taught,  what  the  presentation  mechanism  will 
be,  and  evaluating  the  effects  of  each  phase  of  the  process.  It 1 

can^hardlv  fluTT0  38  3  d°tenninanC  of  content.  Its  proper  application 
-*  «*“  has 
unconventional  clusters  of  instructional  material  may  be  used  for  a  uniform 

the  “same  °f  Wh°m  iS  bein*  PreP^ed  to  perform 

•vW--s^vjto'the  sr  a  r  anCy  !  achleved  by  directing  instruction 
Vhlr  k l'J  .  student  and  to  what  the  student  will  use  on  the  iob 
thereby  assuring  relevance  and  efficiency,  precluding  oversights  and 
adapting  instruction  to  the  individual.  gnts,  d 

a  rt  ?Uring  the  past  20  years*  mi,n>’  attempts  have  been  made  to  cod’fv  a 
t  nit  ve  technology  for  the  systems  approach  to  training.  Earlv  efforts 

e"rlv^ thfM  bY  HumRR0  behalf  °<  tl>«  ».S.  Amy  i„  tho 

approach  „  the 

start,  »lth  the  development  of  a  man-necL.’sJsL^  ™IH"'iI52lPr0?S  Vhlck 
tha  .  a  job  nodal  i.  developed  vhich  then  leads  to  both  the  spocmc.d 
the^rof^r  ^  ^  required  for  adequate  performance  of  the  job  and 

lilllE  £  r'r  deVel°Pment-  The  P^ficiency  test  measures  Le 
lob  If!V  student  to  perform  actual  job  tasks,  thereby  assessing  the 

knowlldee  ^  a  St"dcnt‘  Frora  che  specification  of  skills  and 
knowledge  one  may  determine  the  instructional  objectives,  which  is  to  sav 
those  specific  requirements  for  an  instructional  program.  Onct  the 
instructional  objectives  have  been  determined,  then  a  training  program 

program?°nStrUCted'  ^  S6Venth  Step  ls  the  evaluation  of  the  draining 

One  of  the  more  notable  of  recent  systems  approach  efforts  is  the 

[5  6  JTJr  raSif°r  InatruCtlonal  Systems  Development  (IPISD)  Model 

lb, n,7, 8, 9J.  This  model  was  prepared  by  the  Center  for  Educational 


Technology  at  Florida  State  University  under  contract  with  the  Inter¬ 
service  Committee  for  Instructional  Systems  and  Development,  involving 
the  Army,  Navy,  Air  Force  and  the  Marine  Corps.  The  IPISD  contains 
standardized  rationale,  terminology,  and  basic  concepts  of  instructional 
systems.  These  have  evolved  by  developing  and  recording  the  results  of 
efforts  in  theoretical  and  guidance  materials  required  for  actually  per¬ 
forming  instructional  systems  development.  Prior  to  this  effort,  the  Air 
Force  had  undertaken  a  large  activity  to  develop,  define,  and  record  a 
definitive  technology  for  instructional  systems  development  [10],  and  the 
Army  had  embarked  on  an  ambitious  five-year  program  to  systems  engineer 
all  of  its  training  courses  [11] .  Some  of  the  development  of  the  systems 
approach  to  training  has  gone  on  outside  the  Services,  particularly  in 
industry  [12,13,14].  In  addition,  Mager  [15]  and  others  such  as  Glaser 
[16],  Ammerman  [17],  Krathwohl  [18],  Bloom  [19],  Melching  [20],  Gagne  [21], 
Esbensen  [22],  Bond  [23],  and  Butts  [24],  to  mention  only  a  few,  have 
made  significant  contributions  to  systems  approach  models  through  their 
research  in  the  development  of  behavioral  objectives  and  sequencing  of 
instruction.  In  the  Navy,  much  of  the  work  dealing  with  the  systems 
approach  has  been  carried  out  by  USNTEC  with  reference  to  simulation 
(e.g.,  [25,26]).  In  addition,  the  Navy  has  initiated  several  major 
efforts  related  to  training  systems  design  of  a  more  general  nature  [27, 
28,29]. 

The  IPISD  model  shows  promise  as  a  useful  tool  in  instructional 
system  development  activities  and  is  presently  undergoing  preliminary  field 
evaluation.  The  model  consists  of  five  major  phases  which  can  be  conceived 
under  the  ADDIC  rubric: 

A  analyze 
D  design 
I)  develop 
_I  implement 
control 

Figure  1  is  a  breakdown  of  the  five  phases  into  more  detailed  activities 
(blocks)  comprising  each  phase. 

PROBLEMS  OF  IMPLEMENTING  INSTRUCTIONAL  SYSTEM  DEVELOPMENT  MODELS 

Early  applications  of  the  systems  approach  to  training  were  accomplished 
by  expert  training  developers.  In  the  1960s,  the  possibility  of  having 
laymen  use  these  models  to  achieve  the  success  of  the  experts,  by  imitating 
their  actions,  was  explored.  The  use  of  an  ISD  manual  by  existing  military 
personnel  with  little  or  no  experience  in  training  program  design  may  cost 
a  fraction  of  the  cost  of  hiring  or  contracting  experts  to  do  the  development. 
Even  so,  the  cost/effectiveness  of  the  model  will  still  depend  on  the 
effectiveness  of  the  model,  or  tools,  in  enabling  laymen  to  produce 
effective  instruction. 
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THE  BLOCKS  IN  EACH  PHASE  ARE 


I  i 

►  ANALYZE 
JOB 


I  2 

SELECT 
(ASKS 
E  UNCTIONS 


I  3 

CONSTRUCT 
►  JOB 

PERFORMANCE 

MEASURES 


I  4 

ANALYZE 

'existing 

COURSES 


15 

.SELECT 

'instructional 

setting 


-1 

.  n  i 

DEVELOP 

1 

~  OBJECTIVES 

n  2 

DEVELOP 

TESTS 


n  3 

describe 

ENTRY 

BEHAVIOR 


□  4 

OETERMINE 
SEOUENCE  8, 
STRUCTURE 


m  2 

in  1 

SPECIFY 

m  3 

^  REVIEW/SELECT 
Y  EXISTING 

materials 

^  SPECIFY 

-  k  LEARNING 

^  EVENTS/ 
ACTIVITIES 

^  INSTRUCTION 
^  MANAGEMENT 
^  PLANS. 
DELIVERY 
SYSTEM 

l  in  i 

m  Df VfcLOP 
^  INS r HUCT ION 

m  5 

VALIDATE 

INSTRUCTION 


m  i 

*  IMPl  EMENT 

.  nr  ? 

-  k  INSTRUCTIONAL 

^ CONDUCT 

*  MANAGEMENT 

^  INST  HUCTION 

plan 

In  the  past  few  years,  . problems  with  attempts  at  implementing  ISD 
models  by  laymen  have  surfaced.  Montmerlo  [30]  conducted  a  comprehensive 
review  of  ISD  state-of-art  and  problems  of  implementation.  His  conclusion 
was  that  "available  ISD-type  methodologies  will  not  allow  the  layman  to  be 
as  successful  as  the  expert."  (The  particular  IPISD  manuals  with  which  we 
are  concerned,  however,  were  not  a  part  of  this  review.) 

Montmerlo  cites  the  paper  by  Rieketson,  Schulz  and  Wright  [31]  as  the 
"most  significant  article  concerning  the  problems  of  ISD,"  because  it 
represents  "the  only  empirical  evaluation  of  an  ISD-type  methodology . " 
Rieketson,  Schulz,  and  Wright  studied  the  CONARC  RF.C.  350-100-1  and  its 
implementation  by  Army  instructional  developers.  Although  IPISD  is  Intended 
to  be  a  considerable  improvement  over  350-100-1,  many  of  the  same  problems 
do  apply.  For  example,  Rieketson,  et  al.,  found  that  "High  rates  of 
personnel  turnover  within  some  curriculum  development  groups  have  resulted 
in  a  general  reduction  of  systems  engineering  program  productivity." 

Assuming  this  to  be  a  continuing  reality  in  military  instructional  develop¬ 
ment,  the  need  becomes  evident  to  provide  authoring  aids  which  can  be 
quickly  learned  by  new  developers.  The  study  found,  among  other  things, 
that  developers  tended  to  develop  training  programs  that  employed  the 
same  techniques  with  which  they  had  been  taught,  since  they  did  not  have 
the  ability  to  assess  other  training  techniques  and  equipment. 

IPISD  and  other  ISD  manuals  are  intended  to  have  general  applicability. 
However,  it  has  been  clearly  recognized  by  many  experts  that  the  same 
methodologies  cannot  be  applied  to  the  universe  of  training  problems.  The 
literature  on  task  analysis,  for  example,  contains  a  number  of  articles  on 
the  impossibility  of  using  the  same  method  for  all  tasks  [32,33, 34 ] . 

The  IPISD  Executive  Summary  &  Model  [9]  also  emphasizes  the  need  for 
different  methodologies  in  the  statement,  "The  extent  that  one  used  the 
interview  method,  the  observation  method,  or  the  occupational  survey  method 
depends  on  the  nature  of  the  job  being  analyzed,  the  job  data  already 
available,  and  the  availability  of  analyses  resources."  [9] 

While  the  "what  to  do"  may  remain  relatively  constant  across  training 
problems,  the  "how  to  do  it"  may  vary  enormously.  This  again  is  why  the 
instructional  systems  designer  needs  a  wealth  of  aids  to  refer  to  in 
dealing  with  a  specific  training  problem  in  a  specific  subject  area. 

While  the  IPISD  manuals  do  provide  far  more  references  to  the  literature 
than  previous  manuals  did,  they  do  not  provide  specific  "how  to  do  it" 
guidance  for  specific  design  and  development  tasks.  [9t  p,  124] 

Another  major  problem  area  of  ISD  implementation  in  general  is  the 
management  of  the  Instructional  development  process.  For  example,  when  a 
change  is  made  In  the  conditions  of  a  particular  test  item,  this  has  many 
ramifications  backward  and  forward  in  the  ISD  process.  The  management  of 
these  changes,  including  the  communication  among  various  members  of  the 
IPISD  team,  is  complex  and  usually  requires  some  management  aids. 

Discussions  with  training  staff  at  Ft.  Belvoir  provided  us  with  practical 
evidence  that  it  Is  In  the  area  of  management  of  the  ISD  process  that 
major  problems  continue  to  be  found. 


Other  problems  of  using  IPISD  relate  to  the  background  skills  of  the 
team  leader  and  of  the  members  of  the  development  team.  This  is  pointed 
out  in  the  IPISD  Executive  Summary  [9],  in  the  Montmerlo  study  [30],  and 
in  many  other  sources  [35,36,37]. 

The  IPISD  model  is  advanced  over  other  systems  approach  models  in 
providing  guidance  to  the  training  manager.  However,  the  IPISD  manuals 
are  not  presently  intended  to  provide  specific  procedures  for  every  instruc¬ 
tional  situation  that  can  be  encountered.  Some  situations  are  now  covered 
only  by  the  general  principles  underlying  the  Model.  If  IPISD  is  to  have 
a  "fair  chance"  of  being  accepted  by  training  managers,  it  is  essential 
that  tools  and  author  aids  be  developed  that  will  permit  training  personnel 
to  readily  and  effectively  translate  recommended  IPISD  procedures  into 
meaningful  instructional  products.  This  rationale  forms  the  basis  for 
the  initiation  of  the  present  project. 


THE  NEED  FOR  AUTHOR  AIDS 

Author  aids  are  any  products  used  in  accomplishing  one  or  more  steps 
of  tl\e  IPISD  procedures.  Under  this  definition,  thousands  of  guidebooks, 
research  studies,  texts,  professional  articles,  and  technical  reports 
could  be  considered  as  aids.  If  an  instructional  system  designer  were 
familiar  with  the  full  range  of  aids  available,  lie  would  tn  fact  bo  an 
expert  in  the  field  of  IPISD  and  therefore  not  be  a  subject  of  our  concern. 
The  problem  for  the  author  (any  member  of  the  development  team),  is  to 
know  what  aids  to  use  when,  to  know  they  exist,  to  have  access  to  them  in 
a  timely  way,  and  to  have  some  facility  and  Judgment  in  their  application. 

In  this  effort  two  of  the  blocks  of  the  IPISD  model  were  selected 
for  evaluation.  Block  II. 2,  Develop  Tests,  and  Block  III. 4,  Develop 
Instruction.  The  first  block,  II. 2,  was  required  in  the  RFQ.  Our 
choice  of  Block  III. 4  is  based  on  a  number  of  mutually  supportive 
general  and  specific  reasons.  Many  leading  Instructional  technologists 
and  designers  have  concluded  from  their  experience  that  the  Development 
Phase  of  instructional  preparation  is  the  significant  component  of  the 
systematic  approach  to  producing  quality  instruction.  It  is  expensive, 
time  consuming,  critical,  and  requires  specialized  capabilities.  Van  Pelt 
and  Rich  [38],  for  example,  speak  from  experience  in  the  Army  training 
environment:  "There  is  no  question  that  much  time  is  wasted  by  writers 
casting  about  for  a  reasonable  set  of  guidelines  to  follow  that  will  result 
in  lessons  requiring  a  minimum  of  editing  and  revision."  For  the  Navv, 
Aagard  and  Brabv  [39]  have  emphasized  the  need  for  an  algorithmic  approach 
to  translate  basic  learning  events  into  instruct ionallv  meaningful  task 
categories  ".  .  .  in  a  manner  that  emphasizes  the  flow  of  events  and  the 
combining  and  sequencing  of  learning  guidelines  in  the  design  of  a  training 
program.  .  ."  (p.  7).  In  the  civilian  sector,  Lipson  [40]  has  stressed  the 
need  for  ".  .  .  Increased  Investment  in  development  of  instructional 
materials."  The  "homemade"  variety  doesn’t  have  ".  .  .  the  qualities  of 
craftsmanship,  artistry,  nor  the  proper  incorporation  of  what  is  known 
about  effective  instructional  design  to  be  widely  used."  Industrial 
developers  of  CAI  (Simonson  and  Renshaw  [41])  have  stressed  that  ".  .  . 
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the  cost  of  lesson  preparation  actually  more  than  doubles  the  projected 
cost  of  an  hour  of  CAI  and  cannot  be  ignored."  Recent  analyses  (formal 
and  informal)  by  the  Training  and  Evaluation  Group  (TAEG)l  support  this 
assertion  for  the  general  case  of  individualized  instruction  that  involves 
a  systematic  approach  to  the  development  of  materials. 

On  a  specific  and  practical  basis  we  have  learned  from  the  Curriculum 
Development  Personnel  at  the  USAES  that  they  have  experienced  the  most 
difficulty  in  using  IPISD  with  Block  III. 4,  Develop  Instruction.  A 
comparison  of  test  construction  to  general  development  of  instruction 
suggests  that  in  developing  tests  (IPISD  Block  II. 2),  authors  need  help 
in  performing  such  activities  as  the  following: 

•  Developing  test  items  that  actually  test  the  Terminal  Learning 
Objectives  (TLO) ,  Learning  Objectives  (LO)  ,  and  Lesson  Steps  (LS). 

•  Constructing  hands-on  scorable  units  of  Skill  Qualification 
Tests  (SQTs) . 

•  Developing  scoring  procedures. 

•  Writing  test  items  that  will  help  identify  bad  instruction. 

•  Devising  test  items  that  will  support  remediation  strategies. 

•  Generating  test  items  and  alternative  forms  of  items. 

•  Managing  the  test  development  process,  e.g.,  have  all  TLOs  been 
tested? 

•  Obtaining  reliability  measures  on  test  items. 

•  Determining  the  validity  of  test  items. 

In  Block  III. 4,  Develop  Instruction,  a  variety  of  aids  are  needed. 
Some  of  the  activities  and  decisions  which  require  support  include: 

•  Ensuring  reading  level  is  appropriate  to  the  audience. 

•  Deciding  what  kind  of  drill  and  practice  is  needed,  and  how  much, 
for  a  given  task. 

•  Deciding  how  the  student  will  be  able  to  obtain  additional  help. 

•  Determining  the  nature,  frequency,  and  type  of  feedback  to  provide 
to  the  student  during  the  instruction. 


^Personnel  Communication,  Dr.  Richard  Braby  of  the  Training  and  Evaluation 
Group  (TAEC.) . 
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As  an  overriding  concern,  aids  are  needed  to  help  reduce  the  time  it  takes 
to  develop  quality  instruction  and  tests,  and  to  make  the  development 
process  as  efficient  as  possible. 


A  wide  variety  of  aids  exist.  Recently  Logan  142]  completed  a  survey 
of  existing  tools/procedures  which  could  be  used  by  instructional  developers 
in  conjunction  with  the  IPISD.  The  results  of  tills  survey  indicated  that 
aids  exist  for  a  number  of  IPISD  components.  Unfortunately,  a  considerable 
technical  background  and  level  of  expertise  is  required  for  their  use. 

Thus,  even  when  aids  are  available,  there  remains  the  problem  of  using 
them  without  imposing  an  undue  burden  on  the  author. 

The  existence  of  these  aids  testifies  to  the  recognition  that  they  are 
needed.  However,  in  the  case  of  author  aids  it  is  not  the  variety  and 
quantity  in  the  universe  that  counts.  What  matters  is  that  the  appropriate 
aid  be  available,  easy  to  use  and  accessible  at  the  right  time  and  place. 

The  majority  of  existing  materials  are  more  of  the  "what  to  do"  nature 
than  the  "how  to  do  it"  variety.  There  are  a  number  of  general  resource 
guidelines  already  available  to  aid  systematic  development  of  instruction. 
The  following  discussion  of  these  resources  is  illustrative  and  not  intended 
to  be  exhaustive  of  the  field.  Because  they  are  general,  many  of  these 
handbooks,  manuals,  etc.,  are  difficult  to  categorize  in  terms  of  specific 
single  phases  of  IPISD.  However,  many  examples  can  be  categorized  as  giving 
guidance  primarily  in  Analysis,  Design,  Development,  Implementation,  or 
Control.  For  instance,  Harless  [43]  emphasizes  the  importance  of  "front- 
end  analysis"  to  solve  human  performance  problems.  For  his  target  audience, 
primarily  management  personnel,  he  recommends  defining  the  problem,  inves¬ 
tigating  its  characteristics  and  studying  alternative  solutions  and  their 
costs  before  making  any  decisions.  Mcknight's  work  (44]  on  tailoring 
military  training  by  systems  and  job  analysis  provides  another  useful 
resource  for  the  Analysis  Phase.  The  recently  produced  Marine  Corps 
training  guide  for  task  analysis  [45]  is  another  such  aid. 

Mager's  techniques  [46]  are  classic  as  aids  to  stiuplating  precise 
instructional  objectives  and  thus  fit  into  the  Design  phase  of  IPISD. 

The  TAEG  approach  to  categorizing  instructional  tasks  according  to 
particular  learning  algorithms  [39]  can  in  some  instances  be  useful  for 
design  purposes. 

The  brief  book  by  Pipe  [471  for  beginners  falls  between  Design  and 
Development .  It  describes  the  systematic  procedures  necessary  to  begin 
writing  programs.  He  neglects  the  details  such  as  frame  writing  or 
program  format,  concentrating  on  the  "practical"  Issues  of  steps  that 
precede  writing,  testing  programs,  and  avoiding  pitfalls.  Drumheller's 
handbook  [48]  is  even  broader.  In  his  guide  of  curriculum  design  for 
individualized  instruction  he  has  highlighted  the  need  for  materials  to 
"have  built-in  comprehensiveness."  Ills  systems  approach  provides  guide¬ 
lines  in  the  form  of  a  detailed  model  for  curriculum  design.  It  includes 
defining  objectives,  analyzing  sub-objectives  and  integrating  them  into 
the  learning  experiences.  Wong  and  Raulerson's  [491  guide  spans  all  the 
steps  of  IPISD  in  brief  but  they  spend  more  time  and  give  more  detailed 
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aid  for  Design  and  Development.  In  the  latter,  they  provide  guidance  for 
media  selection  based  on  variations  in  stimulus  requirements  of  the  learning 
tasks.  Their  selection  guide  is  useful,  and  while  not  apparently  theo¬ 
retically  derived  from  learning  principles  as  is  Briggs'  approach  [50],  it 
may  be  more  helpful  to  the  layman.  As  with  most  such  selection  guides, 
the  user  still  must  choose  from  two  or  three  potentially  equivalent  alter¬ 
natives.  Wong  and  Raulerson's  model  [49]  is  based  in  general  on  a  strict 
sequential  view  of  learning. 

A  number  of  published  texts  are  available  as  aids  for  various  parts 
of  the  Development  Phase  of  IPISD.  Markle  [51]  has  discussed  the  construc¬ 
tion,  format,  and  sequencing  of  the  frames  in  her  texts,  both  of  which  are 
programmed. Sperry  [52]  with  its  Instructional  Program  Development  Workbook, 
has  developed  and  used  comprehensive  plans  for  a  workshop  (along  with  a 
workbook)  on  instructional  program  development.  Its  structure  follows  the 
philosophy  of  mathetics  that  includes  demonstration,  prompt,  and  release 
exercises.  Espech  and  Williams  [53]  in  their  Handbook  for  authors  of 
programmed  instruction  describe  the  process  of  constructing  programs  with 
major  emphasis  on  editing,  testing,  and  analysis.  The  finished  product  is 
then  assured  of  being  a  "packaged  change  of  behavior."  Hawkridge,  Campeau, 
and  Trickett  [54]  provide  a  rather  unique  resource  to  help  the  evaluator 
prepare  his  reports.  While  it  is  written  towards  a  school  system  audience, 
its  clear,  concise  approach  should  make  it  usable  in  a  military  context. 

More  recently,  and  still  under  development,  Hillelsohn  is  employing  a 
programmed  instruction  approach  to  creating  and  managing  computer-based 
learning  materials  [55].  This  effort  is  expected  to  provide  an  additional 
means  for  implementing  several  components  of  the  IPISD  model. 

The  most  relevant  example  for  the  Army  as  an  aid  to  Implementation  of 
instruction  is  the  military  training  manual,  FM  21-6  [56].  Its  format  is 
readable  and  comprehensive,  replete  with  examples  for  the  instructors. 

Not  a  great  deal  of  useful  materials  is  available  for  the  Control 
phase  per  se.  However,  Cogan's  case  study  approach  [57]  is  illustrative 
of  such  aids  which  could  be  a  useful  resource  to  training  managers. 

Having  noted  some  of  the  available  resources,  certain  caveats  are  in 
order.  Rather  than  providing  actual  help  in  performing  the  authoring  work, 
or  even  detailed  "how  to  do  it"  guidance,  most  of  the  existing  author 
aids  may  serve  simply  to  reinforce  or  broaden  the  guidance  provided  in 
IPISD  manuals.  Aagard  and  Braby  [39]  very  carefully  note  the  practical 
limitations  of  the  use  for  their  algorithms  and  guidelines  to  general 
approaches.  "The  task  categories  and  related  guidelines  are  not  at  a 
level  that  will  accommodate  any  training  setting."  Briggs'  handbook  [50] 
may  also  serve  such  a  broad  guidance  function.  Care  should  be  exercised 
in  the  selection  of  aids  to  be  integrated  with  IPISD,  so  that 
IPISD  authors  are  not  confronted  with  a  confusion  of  different  yet 
similar  models,  sets  of  jargon,  procedures  or  forms.  Existing  general 
manuals  differ  from  one  another  in  that  they: 


•  Include  different  steps  or  different  names  for  steps. 

•  Include  different  methods  of  accomplishing  each  step. 

•  Provide  different  levels  of  specificity  in  the  detail  included 

under  each  step. 

•  Provide  different  formats  for  reporting  the  work  accomplished 
under  each  step. 

Thus,  to  provide  the  most  efficient  aids  to  IPISD  developers,  the  guidance 
found  in  some  of  these  manuals  and  guidebooks  needs  to  be  translated  and 
integrated  into  the  IPISD  framework,  rather  than  referred  to  in  its  source 
form. 


One  example  of  a  useful  "how  to  do  it"  guide  is  the  Guidebook  for 
Developing  Criterion-Referenced  Tests  [58  ]  •  To  make  the  guidance  in  this 
book  readily  accessible  and  useful  to  the  IPISD  author,  the  ideas  need  to 
be  integrated  into  the  IPISD  framework.  Another  important  aid  in  the 
testing  area  is  the  recent  Manual  for  Developing  Skill  Qualification  [59] 
and  the  Procedures  for  Validating  Skill  Qualification  Tests  [60]. 

In  some  cases,  the  author  needs  actual  assistance  in  performing  an 
activity,  rather  than  simply  how  to  do  it  information.  For  example, 
automated  readability  indices  can  take  some  of  the  workload  from  the  author. 

Some  aids  are  specific  to  a  particular  method  or  theory  of  instruction. 
The  layman  author  needs  some  basis  for  using  that  particular  approach  or 
method,  and  needs  to  know  that  it  is  one  of  several  alternatives.  Thus, 
for  example,  Markle's  [51]  texts  might  be  relevant  and  useful  to  an  author 
who  has  decided  to  follow  her  particular  approach  to  programmed  instruction. 
Alternatively,  Sperry's  comprehensive  plans  for  a  workshop  and  workbook 
[52]  on  instructional  program  development  as  noted  earlier  follows  the 
philosophy  of  mathetics. 

In  the  more  specialized  areas  of  computer-based  instructional  develop¬ 
ment,  an  array  of  automated  aids  have  been  produced.  The  TICCIT  project 
f6ll»  for  example,  uses  highly  structured  forms  for  text  preparation,  and 
highly  procedural ized  production  techniques  for  authoring  teams.  Similarly, 
Project  IMPACT  [62-68]  developed  standard  formatting  aids  for  authors, 
sophisticated  techniques  for  logic  and  text  separation,  instructional 
management,  etc.  Again,  these  aids  are  highly  specific  to  a  particular 

Instructional  strategy  or  method  and  (particularly  in  the  case  of  TICCIT) 
frequently  constrained  by  system  hardware  and  software  constraints. 

The  array  of  automated  aids  for  authors  of  computer-based  materials 
includes:  special  programming  languages,  test  item  generators,  scoring 
algorithms,  recordkeeping  facilities,  objectives  data  banks,  text  editors, 
graphics  aids,  student  response  analysis  algorithms,  data  analysis  routines, 
statistical  subroutine  packages. 


Among  the  major  problems  for  users  of  computer-administered  instruction 
(CAI)  is  the  high  cost  of  developing  quality  instructional  material.  The 
problem  is  exacerbated  by  the  fact  that  new  CAI  programmers  frequently 
require  several  months  of  training  before  they  are  able  to  produce  quality 
instruction  within  an  acceptable  time  frame.  These  individuals  view  their 
role  as  that  of  subject-matter  experts  and  educators  rather  than  programmers. 
To  satisfy  this  audience,  authoring  aids  are  needed  that  will  permit 
educators  to  rapidly  develop  quality  instruction  without  extensive  CAI 
language  training. 

As  a  first  step  in  meeting  this  need,  and  a  forerunner  of  the  present 
research,  HumRRO  completed  the  development  of  sets  of  author  aids  called 
MONIFORMS  [6/, 68].  These  aids  assist  an  author  in  generating  question-and- 
answer  type  practice  items.  The  author  interacts  with  the  computer  which 
leads  the  author  step-by-step  in  the  creation  of  items,  answers,  feedback, 
remediation,  etc.  The  resulting  practice  items  can  then  become  part  of 
either  an  on-line  or  off-line  course  of  instruction.  The  computer  dialog 
can  also  be  used  off-line  in  the  form  of  a  checklist  for  the  author. 

MONIFORMS  were  developed  specifically  for  the  PLATO  IV  system  TUTOR 
language.  However,  the  concept  of  programming  templates  which  permit 
authors  with  limited  programming  experience  to  create  test  and  lesson 
materials  has  wider  application.  While  MONIFORMS  are  a  valuable  first  step, 
there  still  existed  a  need  for  more  advanced  author  aids.  Preliminary  study 
at  HumRRO  indicated  the  feasibility  of  developing  author  aids  which  through 
interrogation  of  the  course  author  would  automatically  convert  lesson  content 
and  structure  into  executable  program  code.  Therefore,  the  author  would 
require  no  previous  programming  experience  and  thus  make  the  aids  much 
easier  to  use.  This  concept  formed  the  basis  of  the  approach  for  the  current 
effort. 

In  summary,  with  respect  to  authoring  aids: 

1.  There  exists  a  very  rich  array  of  a  wide  variety  of  materials, 
handbooks,  guides,  and  automated  aids  which  could  serve  to  help  in  the 
IPISD  process. 

2.  There  exists  a  very  real  need  for  these  aids. 

3.  The  selection  of  specific  assistance  to  be  integrated  into  the 
framework  of  IPISD  is  a  task  that  is  yet  to  be  completed. 


EVALUATION 

In  the  previous  sections,  we  have  discussed  the  need  for  specific 
aids  or  tools  which  can  assist  authors  to  apply  the  IPISD  process  in  the 
preparation  of  instruction.  In  developing  and  tailoring  aids  for  authors, 
we  are  in  effect  developing  instruction — for  authors.  Once  any  instruc¬ 
tional  product  is  under  development,  a  continuous  process  of  evaluation  and 
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revision  needs  to  take  place.  This  evaluative  process  should  also  be 
applied  to  the  development  of  author  aids — an  approach  we  have  taken  in 
the  present  research  effort.  The  following  discussion  of  evaluation  provides 
background  and  rationale  for  our  approach  to  evaluation. 

Evaluation  Defined 

Evaluation  is  the  process  of  delineating,  obtaining,  and  providing 
useful  information  in  judging  decision  alternatives  [69].  It  is  an  action- 
related  process  which  has  as  its  major  characteristic  the  determination  of 
value,  worth,  or  merit.  The  evaluation  process  is  conceived  as  continuing 
rather  than  as  having  a  discrete  beginning  or  ending.  Evaluation  should 
facilitate  the  continuous  improvement  of  a  program.  It  should  stimulate, 
not  stifle,  instructional  development. 

Evaluation  procedures  may  be  categorized  as  formative  or  summat ive 
[  79] .  Formative  evaluation  is  that  process  which  validates  instruction 
during  on-going  initial  program  development.  The  results  of  this  evalua¬ 
tion  are  acted  upon  immediately  in  program  modification.  In  other  words, 
the  practice  of  conducting  tryouts  of  draft  materials  during  program 
development,  followed  by  measures  which  provide  an  assessment  of  the 
materials  which  lead  to  their  revision  is  referred  to  as  formative  eval¬ 
uation.  Formative  evaluation  is  performed  for  the  purpose  of  diagnosing 
and  correcting  the  weaknesses  of  a  program. 

Anyone  involved  in  the  revision  of  instruction  may  be  engaged  in 
formative  evaluation  (in  the  loosest  sense  of  the  term) .  What  is  presumably 
being  done  is  being  done  because  the  developer  or  someone  else  has  judged 
the  existing  course  as  unsatisfactory.  As  new  materials  are  developed,  they 
are  constantly  being  "evaluated"  as  better  or  worse  than  that  which  already 
exists.  However,  it  is  a  formal  program  of  formative  evaluation  employing 
various  assessment  techniques  which  is  the  keystone  of  the  IPISD  process  and 
which  provides  the  link  between  course  content  and  course  improvement.  By 
explicitly  stating  objectives  and  criteria,  one  can  properly  determine  if 
the  program  is  achieving  its  goals,  or  if  goals  are  to  be  modified. 

Summative  evaluation  is  performed  for  the  purpose  of  assessing  a 
fully  implemented  training  program  with  respect  to  its  ability  to  produce 
graduates  who  can  perform  to  minimum  standards  of  performance.  Also,  the 
evaluation  can  determine  whether  or  not  efficient  and  effective  use  was 
made  of  educational  resources.  Summative  evaluation  should  occur  after 
instructional  development,  improvement,  and  stabilization  of  operational 
and  administrative  activities.  This  may  vary  from  one  training  program 
to  another.  In  some  cases,  training  objectives  may  not  be  measurable 
at  the  desired  time  of  evaluation  because  they  are  either  too  costly  to 
measure  or  are  long-term  objectives. 

Results  of  a  summative  evaluation,  while  of  interest  to  the  developer, 
are  of  primary  concern  to  those  who  will  decide  whether  or  not  a  program 
is  to  be  continued  or  adopted.  Summative  evaluation,  therefore,  provides 
the  basis  for  policy  decisions  that  do  not  necessarily  concern  revision 
of  the  program  or  product  [71,72] . 


The  differences  between  formative  and  summative  evaluations  are  mainly 
in  their  purposes  and  the  timing  of  their  application.  Formative  evaluation 
is  continuous  in  nature  and  serves  to  refine  a  given  program  through  an 
iterative  feedback  process;  summative  evaluation  produces  final  judgments 
concerning  the  degree  to  which  program  objectives  and  goals  have  been 
attained.  The  information  obtained  from  a  summative  evaluation  allows  the 
user  to  judge  whether  a  program  meets  his  needs,  whether  it  should  be  widely 
disseminated,  and  if  alternatives  exist,  which  are  to  be  preferred. 

The  discussion  which  follows  will  concentrate  primarily  upon  format ive 
evaluation  as  the  purpose  of  the  effort  reported  herein  was  to  evaluate 
author  aids  in  their  initial  development  stages. 

Evaluation  Models 

The  formal  distinction  between  formative  and  summative  evaluation  is 
attributed  to  Scriven  [70].  However,  the  purposes  for  which  such  evaluation 
data  are  used  have  been  discussed  for  many  years  in  the  training  and 
education  literature.  Cronbach  [73]  stated  that  "the  greater  service 
evaluation  can  perform  is  to  identify  aspects  of  the  course  where  revision 
is  desirable."  Early  models  of  the  systems  approach  to  training  development 
contain  quality  control  components  which  emphasize  the  need  for  feedback 
for  program  improvement.  Smith  [74]  described  the  purpose  of  a  quality 
control  system,  "...  a  means  for  continuous  monitoring  of  the  quality 
of  the  graduates  and  for  improving  the  training  when  it  is  deficient." 

Quality  control  procedures  are  needed  both  at  the  school  and  in  the 
field.  Information  from  both  locations  must  be  "fed  back"  so  that  the 
instructional  program  can  be  appropriately  adjusted.  Schools  require  two 
types  of  feedback  information.  The  first  type  assesses  the  ability  of  a 
course  graduate  to  perform  acceptably  those  tasks  which  the  instructional 
program  claims  to  teach.  This  type  of  information  assesses  the  ability  of 
the  instructional  program  to  teach  well  whatever  it  is  that  it  claims  to 
teach.  In  most  instances,  this  assessment  can  be  made  at  the  school. 

A  second  type  of  feedback  information  deals  with  the  discrepancies 
between  the  course  graduate  performances  and  field  requirements.  "Relevancy 
control"  information  assesses  whether  or  not  the  instructional  program 
teaches  the  appropriate  subjects  or  tasks,  and  whether  or  not  the  student 
can  transfer  these  capabilities  to  the  field.  Also,  this  feedback  should 
provide  information  dealing  with  changing  field  requirements  and  with  more 
precise  descriptions  of  job  activities. 

Baker  and  Alkin  [75]  point  out  that  the  evaluative  process  was  an 
integral  part  of  programmed  instruction  development  which  antedated  the 
surge  of  interest  in  formative  evaluation  during  the  past  decade. 

Recent  models  of  the  formative  evaluative  process  include  those  of 
Stake  ["‘6],  Scriven  [77],  Stufflebeam,  et  al.  [69],  Sanders  and  Cunningham 
[78],  and  Rippey  [79].  Scriven  [77]  feels  that  it  is  best  if  formative 
evaluation  is  performed  by  someone  other  than  the  developer.  Scriven 
calls  his  approach  "goal  free  evaluation"  which  calls  for  the  evaluator 


to  assess  the  actual  effects  of  the  program.  The  evaluator  operates  with¬ 
out  knowledge  of  the  purposes,  goals,  or  objectives  of  the  program 
developers.  Another  model  is  described  by  Stake  [76]  as  "responsive 
evaluation"  which  calls  for  the  evaluator  to  be  external  to  an  instructional 
development  activity,  and  therefore  to  have  a  certain  independence  and 
objectivity  that  is  presumed  not  to  be  present  in  an  internal  evaluator. 

The  Stake  model  provides  a  process  evaluation  strategy  which  contains  a 
two-stage  procedure:  the  first  determines  congruence  between  what  is 
intended  and  what  is  actually  observed  (that  is,  discrepancies  from 
program  specifications),  and  the  second  with  making  sure  the  program  has 
the  type  and  quality  of  components  implied  by  its  objectives. 

Less  dependence  is  placed  on  the  external  evaluator  by  Stuff lebeam 
[691.  As  this  model  emphasizes  the  need  for  evaluation  data  to  serve 
decision-making  purposes  in  a  timely  manner,  it  permits  the  evaluator  to  be 
part  of  the  development  team.  The  "process"  evaluation  component  calls 
for  provision  of  feedback  continuously  during  program  implementation.  In 
a  similar  framework,  Sanders  and  Cunningham  [78]  identify  four  stages  of 
the  formative  process.  The  first  is  called  the  predevelopmental  stage, 
which  seeks  to  identify  needs.  The  second  stage  is  called  evaluation  of 
objectives  in  which  one  develops,  revises,  and  clarifies  objectives.  The 
third  stage  is  called  interim  evaluation,  and  seeks  to  evaluate  each  piece 
of  the  instruction  as  it  is  developed.  The  final  stage  is  called  product 
evaluation,  in  which  the  program  as  a  whole  is  evaluated,  after  which  it 
may  be  recycled  for  further  development. 

Churchman,  et  al .  [80]  discuss  the  question  of  whether  to  use  internal 
or  external  formative  evaluators.  They  make  the  point  (with  which  we  agree) 
that  in  practice  the  formative  evaluator  will  become  so  involved  in  the 
program  that  the  objectivity  expected  from  an  external  evaluator  will  be  of 
little  significance  during  the  formative  process. 

Transactional  Evaluation  (Rippey  [79];  Seidel  [81])  differs  from  other 
evaluation  models  in  that  it  focuses  on  the  effects  of  perceptions  of  project 
team  members  and  the  user  population.  Its  usefulness  in  forr-'tive  evalua¬ 
tion  comes  from  its  emphasis  on  making  explicit  the  relationships,  roles, 
problems  and  possible  solutions  as  perceived  by  developers  and  potential 
users  of  the  instruction.  The  formal  involvement  of  these  people  in  clari¬ 
fying  the  goals  and  objectives  of  a  given  program  contribute  to  improvement 
during  its  early  formative  stages. 

Formative  Evaluation  Techniques 

The  same  measurement  techniques  and  procedures  may  be  employed  in 
formative  and  summative  evaluation.  It  is  the  purpose  to  which  the  evalua¬ 
tive  effort  is  put  and  the  time  when  it  occurs  that  distinguishes  between 
the  two  types  of  evaluations. 

The  application  of  experimental  design  to  evaluation  problems  conflicts 
with  the  princ iple  that evaluat ion  should  facilitate  the  continual  improvement 
of  a  program.  Experimental  design  prevents  rather  than  promotes  changes  in 
the  treatment  because  treatments  cannot  be  altered  in  process  if  the  data 


about  differences  between  treatments  are  to  be  unequivocal  182].  The 
experimental  design  type  of  evaluation  is  useful  for  making  summative 
decisions  but  almost  useless  as  a  device  for  making  decisions  during  the 
planning  and  implementation  of  a  project  ] 23] . 

In  formative  evaluation,  the  developer  is  looking  for  what  the 
researcher  often  takes  great  pains  to  avoid.  Instruction  changes  as  a 
function  of  his  activity,  both  as  it  is  being  developed  and  as  it  is 
implemented  in  pilot  or  field  tests.  The  summative  evaluator,  on  the 
other  hand,  as  does  the  researcher,  goes  to  great  lengths  to  hold  the 
program  constant. 

The  choice  of  design  for  a  formative  evaluation  is  a  complicated 
decision  depending  upon  a  number  of  considerations:  cost,  utility, 
practicality,  t  'neralizability,  etc.  Campbell  and  Stanley  [83]  have 
discussed  the  major  considerations  in  the  choice  of  a  design.  The  evaluator 
needs  to  be  concerned  with  replicability  in  that  if  the  effect  of  instruc¬ 
tion  cannot  be  reliably  established,  then,  of  course,  decisions  about  how 
to  make  it  better  are  meaningless. 

The  most  frequently  used  design  in  instructional  evaluations  is  the 
single  group  pre-test/post-test  design  [ 84 ] .  In  this  quasi-experimental 
design,  a  single  group  of  students  is  first  tested  to  determine  how  much 
of  the  criteria  behavior  they  possess,  then  are.  administered  the  instruc¬ 
tion,  then  tested  again.  If  learning  gains  are  demonstrated,  the  product 
developer  concludes  he  has  a  successful  product.  The  problem  with  such  a 
design  is  that  it  allows  many  other  plausible  rival  explanations  for  the 
observed  results.  In  addition,  a  very  serious  limitation  is  the  unreli¬ 
ability  of  change  scores  [85]. 

Pre-  and  post-testing  is  usually  considered  inadequate  for  formative 
purposes.  Continuous  monitoring  permits  correcting  problems  as  they  occur, 
tends  to  increase  the  aspects  of  the  program  that  are  included  in  the 
evaluation,  and  consequently  improves  the  usefulness  of  the  evaluation 
itself. 

One  problem  with  monitoring  is  in  collecting  data  representative  of 
the  performance  of  the  program  such  that  it  is  typical  of  the  full  range 
of  the  intended  usage  of  the  system.  This  collection  of  performance  data 
needs  to  be  done  without  disturbing  the  performance  of  the  system  being 
monitored,  which  is  difficult.  Another  problem  is  assimilating  and  inter¬ 
preting  the  results.  It  is  easy  to  collect  massive  amounts  of  confusing 
dataunless  one  establishes  monitoring  experiments  with  clear  hypotheses  in 
mind  [  86]  . 

The  IPISD  guidance  [87]  recommends  such  tryouts,  as  follows:  "If 
the  student  who  tries  out  the  instruction  experiences  difficulties,  it  may 
be  profitable  to  again  test  out  the  instruction,  after  revisions,  on 
another  student.  Beyond  practical  considerations  of  time  there  is  really 
no  limit  to  the  amount  of  pre-test  tryouts  that  can  be  conducted  until  the 
instruction  is  successful." 


IF  the  task  of  the  formative  evaluutor  is  to  monitor  programs  in  order 
to  provide  evaluation  data  leading  to  improved  instruction,  then  it  is  not 
surprising  that  the  focus  of  most  research  on  formative  evaluation  has  been 
at  the  d;^a-acqu is  it ion/evaluation-ut ilization  juncture  (75]. 

One  interesting  research  question  relates  to  the  selection  of  subjects 
as  a  data  source  for  various  formative  evaluation  efforts.  There  are  those 
(including  IPISD)  which  recommend  that  formative  evaluation  data  are  obtained 
from  single  learners  in  linear  fashion  with  repeated  tryouts.  Essentially 
this  technique  consists  of  placing  the  author  with  a  student  as  he/she  uses 
the  materials.  Ideally,  th^  student  will  help  the  author  locate  ambiguities, 
errors  of  sequence,  and  the  like,  and  allow  the  author  to  test  his  assump¬ 
tions  concerning  the  thinking  processes  which  will  be  employed  by  students 
using  the  materials  [78]. 

An  unpublished  study  by  Robeck  (as  reported  by  Baker  an  Alkin  [75]) 
tested  the  feasibility  of  using  a  single  student  as  the  data  source  for 
formative  evaluation  leading  to  the  revision  of  an  instructional  program. 

The  study  demonstrated  that  observation  of  a  single  student  is  an  economical 
method  for  significantly  improving  instruction.  Aside  from  this  study, 
very  little  research  on  this  technique  has  been  performed.  The  present  state 
of  knowledge  consists  of  a  number  of  conflicting  "tips"  on  how  to  implement 
the  procedures.  Some  recommend  that  high  ability  students  be  used,  others 
recommended  low  ability.  Some  sources  argue  that  students  can  only  clean 
up  semantic  and  syntactic  errors,  while  others  insist  that  the  student  can 
make  more  substantive  suggestions  concerning  sequence,  intended  prerequi¬ 
sites,  etc.  At  present,  even  a  simple  experiment  comparing  the  quality  of 
instructional  products  which  have  and  have  not  used  individual  student 
tryouts  as  part  of  the  development  has  not  yet  been  done. 

As  was  stated  earlier,  a  variety  of  techniques  can  be  used  for  forma¬ 
tive  evaluation.  The  purpose  for  which  the  information  is  gathered  determines 
whether  it  is  formative  or  summative.  The  ultimate  criterion  of  an  instruc¬ 
tional  program,  however,  is  a  change  in  the  behavior  of  students.  Determin¬ 
ation  of  whether  or  not  that  purpose  was  met  requires  a  demonstration  of 
such  changes.  The  IPISD  guidance  states  that  one  needs  to  examine  in  detail 
the  responses  of  the  learners  on  criterion  tests  [87].  A  combination  of 
tests,  observations,  interviews,  and  affective  measures  is  required  to  amass 
the  data  necessary  for  the  formative  evaluation  and  improvement  of  instruction. 
The  specific  techniques  and  approach  used  in  our  project  to  evaluate  the 
author  aids  will  be  discussed  in  Chapter  IV  of  this  report. 
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Chapter  II 
PURPOSE 


The  purpose  of  the  research  effort  described  herein  was  to  conduct  a 
development  and  feasibility  demonstration  of  on-line,  query-based  author 
aids.  The  research  was  designed  to  include  author  aids  for  Blocks  II. 2 
(Develop  Tests)  and  III. 4  (Develop  Instruction)  of  the  Interservice 
Procedures  for  Instructional  Systems  Development  (IPISD). 

Specifically,  the  activities  of  the  project  were  to  result  in  author 
aids  which: 

•  Are  suitable  for  creation  of  both  on-line  and  off-line  instruction. 

•  Are  generalizable  for  differing  subject  matter  areas. 

•  Are  documented  in  a  flowchart  form  to  permit  timely  conversion  as 
appropriate  to  other  CAI  systems. 

The  utility  of  the  author  aids  developed  was  to  be  evaluated  and 
revised  as  necessary  with  military  authors/instructors  preparing 
operationally  relevant  instructional  material. 
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Chapter  III 
APPROACH 


The  goal  of  the  project  was  to  construct.  Implement  and  provide  a 
feasibility  test  of  on-line  authoring  aids  which  can  be  integrated  with 
the  IPISD  model.  In  order  to  attain  the  objectives  of  this  project,  the 
approach  taken  was: 

•  User-oriented 

•  Guided  by  the  IPISD  model 

c  Multi-level  in  its  parallel  development/evaluation  activities. 

A  cooperative  working  relationship  was  established  with  instructional 
and  curriculum  development  personnel  of  the  U.S.  Army  Engineer  School 
(USAES) ,  Ft.  Belvoir,  Virginia.  Input  from  USAES  personnel  was  an  Important 
influence  in  the  selection  of  author  aids  which  would  help  the  USAES 
instructional  development  team  to  implement  the  IPISD. 

Author  aids  to  be  developed  in  this  project  were  presented  on  the 
PLATO  IV  computer-assisted  instruction  (CAI)  system.  During  the  course 
of  the  project,  four  PLATO  IV  terminals  were  located  in  the  HumRRO 
laboratory,  Alexandria,  Virginia.  In  addition,  8  terminals  located  at 
Ft.  Belvoir,  Virginia,  were  also  available  during  the  project. 

The  Engineer  Non-Commissioned  Office  Advanced  (ENCOA)  course  was 
selected  for  this  project  in  consultation  with  USAES  curriculum  development 
and  training  personnel  and  with  the  agreement  of  ARI.  Arrangements  were 
made  to  permit  participation  of  four  instructors  (2  NCOs  and  2  Officers) 
who  teach  this  course. 1  The  ENCOA  course  covers  a  wide  range  of  technical 
("hard")  and  soft  skills.  It  was  thought  that  if  authoring  aids  were 
developed  which  would  be  useful  for  handling  instruction  and  testing  of  hard 
skills  (e.g.,  straightforward  mechanical  work)  as  well  as  soft  skills  such 
as  problem-solving,  the  set  of  authoring  aids  would  be  more  applicable  to 
other  courses  and  other  schools  than  if  just  the  hard  skills  were  chosen 
for  the  targeted  materials.  Therefore,  the  subject  matter  selected  for 
this  project  was  a  nine-hour  block  of  instruction  from  the  ENCOA  course, 
covering  such  items  as  field  fortifications  emplacement  construction, 

U.S. /foreign  mine  warfare  doctrine,  and  protective  mining.  The  work 
involved  in  this  section  of  the  course  includes  computational  problem¬ 
solving,  as  well  as  procedural  tasks.  School  personnel  had  defined  training 
objectives  as  a  result  of  previously  applied  systems  engineering  principles. 


AOne  Officer  was  transferred  from  the  USAES  during  the  course  of  the  project. 
Therefore,  only  three  instructors  participated  in  the  research  effort. 
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The  II’ISD  model  was  a  compatible  and  useful  guide  (m  designing  tin’ 
technical  approach  for  the  proposed  project.  Kach  of  the  first  three 
major  procedural  Phases — Analyze,  Design,  and  Develop--were  pertinent  tv' 
the  activities  undertaken  in  this  project.  The  targeted  "students"  In 
this  case  were  the  authors  and  the  instruct  tonal  focus  was  the  authoi  aid. 


The  multi-level  nature  of  the  project  should  he  considered  here, 

MumKKO  personnel  developed  and  evaluated  author  a l vis.  These  author  aids 
were  then  used  by  US  AGS  instructors  to  dew  lop  and  validate  inst  met  ion. 

Thus,  iterative,  parallel  activities  occurred  at  different  levels  in  the 
project,  (luidtng  all  these  efforts  was  the  1PISD  model  itself-  in  particular 
the  first  three  procedural  Phases.  Kor  example,  the  approach  tv'  author 
aid  development  and  validation  drew  its  guidance  specifically  from  1P1SD 
Blocks  I It. A  and  IU.S. 

Initially,  a  set  of  detailed  flowcharts  were  constructed  tv'  describe 
Information  elements  and  features  required  bv  Instructional  developers  in 
performing  the  steps  of  IP1S0  Blocks  11..'  (Develop  Test  si  and  111. A  (Develop 
Instruction).  The  flowcharts  were  designed  to  be  sufficiently  detailed 
and  annotated  for  ready  adaptation  tv'  unv  system  (i.e.,  relatively 
hardware  or  software  independent). 

Inasmuch  as  the  1'l.ATO  IV  system  was  considered  a  research  vehicle 
only,  care  was  taken  to  maximize  hardware  or  software  independence  of  the 
aids.  On-line  author  aids  as  well  as  off-line  versions  were  supplied  tv' 
assist  the  author  in  preparing  inst ruct Iona l  and  test  materials  fo-  either 
CAT  or  non-CAT  delivery  of  instruction. 

The  multi-level  nature  of  the  research  activities  is  clearlv  demon¬ 
strated  by  the  three  levels  of  evaluation  undertaken  in  the  project.  The 
first  level  was  an  iuforma  1  eval uat  ion  of  existing  IP1S0  guidance,  procedures 
and  author  aids.  ItumRJTfi  staff,  as  users  of  this  guidance,  were  the  primary 
source  of  evaluation  data  at  this  level. 

level  2  was  directed  toward  a  formative  evaluation  of  new  author  aids 
and  procedures  developed  specifically  for  on-line  application  tv'  11'IKP 
Blocks  11.2  and  I II. A.  level  1  evaluation  assessed  the  adequacy  of  the 
instructional  materials  created  by  the  military  authors.  These  materials 
were  then  administered  tv'  U.S.  Army  Engineer  School  trainees  who  provided 
an  additional  data  source. 

Revision  activities  occurred  continuously  throughout  the  period  of 
project  performance.  The  purpose  of  these  revisions  was  tv'  assure  maximum 
utility  of  the  flowcharts  and  author  aids  in  implementing  the  1PISD  process. 
The  test  items  and  lesson  material  were  not  revised  as  a  basis  of  trainee 
data,  because  of  t  i nu-  limitations,  but  these  data  were  incorporated  as  part 
of  the  research  conclusion. 

The  project  activities  were  divided  into  four  major  Tasks.  These  were: 

Task  1.  Analysis  and  Determination  of  Required  Author  Aid  Elements 


Task  2. 


Conversion  o£'  Flowcharts  to  Interactive  Program 
Task  3.  Evaluation  of  the  Programmed  Materials 
Task  4.  Revisions 

The  activities  and  accomplishments  in  each  of  the  Tasks  are  described  in 
the  following  chapter. 


Chapter  IV 

PROJECT  ACTIVITIES  AND  ACCOMPLISHMENTS 


TASK  1.  ANALYSIS  AND  DETERMINATION  OF  REQUIRED  AUTHOR  AID  ELEMENTS 

In  Task  1  a  detailed  set  of  flowcharts  was  constructed  which  provide 
instructional  system  designers  with  the  means  of  performing  the  procedures 
called  for  by  IPISD  Blocks  II. 2  (Develop  Tests)  and  III. 4  (Develop 
Instruction).  Activities  during  Task  1  were  conducted  in  two  Phases: 

(1)  expansion  of  IPISD  flowcharts  so  as  to  provide  greater  detail  of  the 
specific  activities  required  for  each  block  (What  To  Do),  and  (2)  selecting, 
identifying,  designing  and  flowcharting  of  author  aids  for  completing  the 
activities  (How  To  Do). 

1 

Develop  What-To-Do  Flowcharts 


The  IPISD  flowcharts  for  Blocks  II. 2  and  III. 4  shown  below  in  Figures 
2  and  3  were  used  as  the  basic  framework  for  the  HumRRO-deve loped  flow¬ 
charts.  The  IPISD  flowcharts  provide  a  broad  description  and  sequencing 
of  necessary  activities.  However,  because  of  their  global  nature,  they 
provide  only  minimal  assistance  to  the  instructional  systems  designer. 

Each  element  of  the  IPISD  flowcharts  was  expanded  into  detailed 
step-by-step  sub-elements  that  must  be  performed  (or  considered)  in 
completing  the  specific  flowchart  block.  With  respect  to  the  IPISD  activity. 
Develop  Tests  (IPISD  Block  II. 2),  the  procedural  steps  described  in  the 
"Guidebook  for  Developing  Criterion  Referenced  Tests"  ( 58 1  were  used 
heavily  in  the  identification  of  the  sub-elements.  Figure  4  is  an  example 
of  how  one  such  IPISD  element,  2.6  (Determine  Scoring  Procedure)  from 
Block  I I. 2  was  expanded  into  sub-elements. 

It  was  found  that  the  activity  descriptions  shown  in  sub-element 
blocks  were  not  always  sufficiently  descriptive  of  the  activities  required 
by  the  block.  Consequently,  it  was  necessary  to  further  flowchart  several 
of  these  sub-element  blocks.  An  example  of  further  flowcharting  of  sub¬ 
element  block  2.6.1  (Determine  Qualitative  Scoring  Procedures)  is  shown  in 
Figure  5.  The  blocks  in  italics  refer  to  blocks  already  flowcharted  in 
IPISD.  All  other  blocks  are  HumRRO  flowcharts.  Blocks  outlined  in  hold- 
face  are  blocks  for  which  author  aids  were  developed  in  the  project.  A 
check  mark  above  a  block  indicates  that  existing  author  aids  have  been 
identified  for  that  block.  The  narrative  on  the  right  of  the  flowchart 
further  clarifies  the  block  and  lists  any  references  to  existing  author 
aids. 


The  product  which  resulted  from  the  Phase  1  activities  is  itself  a 
valuable  author  aid  for  instructional  system  designers.  It  provides  a 
step-by-step  enunciation  of  activities  that  must  be  performed.  The  revised 
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Figure  4 
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Determine  Qualitative 
Scoring  Procedures. 


Determine  whether  partially 
correct  responses  will  be 
evaluated. 


Especially  applicable  in  constructed  response  format 


Determine  scheme  for 
assessing  partially  correct 
answers. 


For  oral  and  constructed  response  (essay)  tests 


Determine  whether  presentation 
of  response,  grammar,  spelling 
or  punctuation  will  affect 
score. 

2.6. 1.3 


Outline  scoring  guidelines  for 
behavioral  checklists  or 
scales. 


Illustrative  Flowchart  Expansion  of  Block  2.6.1 


and  expanded  flowcharts  were  produced  as  off-line  materials.  As  such, 
they  can  be  converted  to  checklists  and  used  as  procedural  guides  by 
designers  of  instruction. 

It  was  not  feasible  within  the  limits  of  this  project  to  produce 
the  fully  expanded  flowcharts  on  the  PLATO  terminal.  However,  the 
information  contained  in  many  of  the  blocks  was  incorporated  into  the  on¬ 
line  author  aids  and  as  such  provide  on-line  guidance  in  accomplishing  the 
objectives  of  each  block. 

Identify  and  Reference  How-To-Do- It  Author  Aids 

In  Phase  I  the  detailed  activities  (sub-elements)  needed  for  developing 
tests  and  instruction  were  defined  and  arranged  into  sequential  order.  In 
Phase  2  each  sub-element  was  examined  to  determine  specific  authoring  aids 
desirable  to  accomplish  the  sub-element.  In  other  words.  Phase  1  describes 
what  must  be  done,  and  Phase  2  defines  author  aids  for  doing  it.  Time 
constraints  did  not  permit  the  development  or  selection  of  author  aids  for 
every  sub-element.  Therefore,  aids  were  provided  for  those  sub-elements 
which  were  identified  as  of  highest  priority  for  potential  users.  The 
selection  was  based  on  such  factors  as: 

1.  Available  HumRRO  expertise  gained  from  previous  experience  in 
author  aid  development; 

2.  A  review  of  the  literature  to  identify  aids  already  available 
for  use;  and 

3.  Opinions  of  instructional  systems  designers  at  USAES  concerning 
aids  they  considered  would  be  helpful  to  them. 

After  identifying  the  author  aids  needed  for  the  IPISD  sub-element 
blocks,  the  next  step  accomplished  in  Phase  2  was  that  of  including 
references  to  the  author  aids  in  the  flowcharts  developed  in  Phase  1.  The 
purpose  of  this  step  was  to  identify  for  users  of  the  flowcharts  those  sub¬ 
elements  for  which  author  aids  were  available  and  to  refer  them  to  a 
reference  (hardcopy  or  on-line)  which  more  fully  detailed  the  specifications 
of  the  aid. 

Throughout  Phases  1  and  2,  care  was  taken  to  assure  that  the  flowcharts 
were  sufficiently  detailed  and  annotated  to  be  of  practical  use  to  instruc¬ 
tional  system  designers  and  would  readily  permit  adaptation  to  any  system, 
i.e.,  be  hardware  or  software  independent. 

Task  1  activities  resulted  in  an  Interim  Report  and  Guide  to  the  Use 
of  Flowcharts  [88].  This  report  contains  flowcharts  providing  detailed 
guidance  on  the  procedural  steps  necessary  for  implementing  IPISD  Blocks 
II. 2  and  III. 4,  and  identifies  sub-elements  of  these  blocks  for  which  author 
aids  were  developed.  In  addition,  the  Interim  Report  includes  a  guide  to 
the  use  of  flowcharts  which  we  felt  is  necessary  inasmuch  as  many  authors 
may  not  be  familiar  with  a  flowchart  format. 


The  flowcharts  prepared  in  Task.  1  are  of  value  from  three  standpoints: 
(1)  They  are  useful  as  tools  for  instructional  systems  designers  in  the 
implementation  of  the  IPISD  process;  (2)  they  may  be  used  as  a  model  for 
detailing  the  processes  covered  in  other  IPISD  Blocks  in  terms  of  level  of 
detail,  style,  and  format;  and  (3)  they  may  be  used  in  the  preparation  of 
on-line  author  aids  on  any  CAI  system. 


TASK  2.  CONVERSION  OF  FLOWCHARTS  TO  INTERACTIVE  PROGRAM 

In  Task  2,  author  aids  identified  in  Task  1  were  developed  for  presen¬ 
tation  on  the  PLATO  IV  computer-assisted  instruction  (CAI)  system.  Inasmuch 
as  the  PLATO  IV  system  was  considered  a  research  vehicle  only,  care  was 
taken  to  insure  that  the  author  aids  developed  could  be  readily  modified 
to  be  hardware  or  software  independent.  Where  possible  the  author  aids 
were  also  created  so  as  to  have  application  for  "off-line"  use. 

In  the  context  of  the  present  project,  an  instructional  system  developer 
(author)  may  be  working  on-line  in  an  interactive  mode  with  a  computer.  In 
this  case,  he  is  termed  an  "on-line  author."  If  an  author  is  not  working 
directly  with  a  computer,  he  is  referred  to  as  an  "off-line  author."  Even 
authors  who  are  developing  CAI  materials  work  in  both  off-line  and  on-line 
modes.  For  example,  some  authors  use  preprinted  CRT  layout  sheets  to  write 
their  text,  and  then  have  clerks  key  the  text  into  the  computer.  Currently 
authors  of  most  military  instruction  typically  work  off-line,  although  they  may 
have  access  to  computer  support  for  such  things  as  test  scoring,  statistical 
item  analysis,  or  other  aids. 

Author  aids  were  developed  to  assist  the  on-line  author  in  preparing 
instructional  and  test  materials  for  both  CAI  and  non-CAl  delivery  of 
instruction.  CAI  was  the  principal  mode  used  in  this  research  because  the 
CAI  mode  provides  opportunity  for  ease  of  gathering  and  analyzing  data 
regarding  both  student  and  author  activities.  Off-line  versions  of  these 
aids  will  assist  off-line  authors  in  preparing  both  CAI  and  non-CAl  materials. 

PLATO  lessons  "inquiryl"  and  "inquiry2"  can  be  thought  of  as  master 
author  aids.  These  lessons  incorporate  the  Individual  author  aids  identified 
in  Task  1.  Lesson  "inquiryl"  deals  with  lesson  development  (IPISD  Block 
III. 4),  and  "inquirv2"  with  test  development  (IPISD  Block  11.2).*  Individual 
author  aids  for  inquiryl  and  2  can  be  roughly  categorized  in  four  different 
classes  which  are  discussed  below.  The  four  categories  for  each  master 
author  aid  are: 


^These  lessons  were  available  on  the  Universitv  of  Illinois  PLATO  IV  CAI 
system  at  the  time  this  effort  was  completed. 
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Lesson  "inquiry!"  (Lesson  Development) 


1.  Tutorial  author  aids  for  Lesson  Development 

2.  Author  aids  for  management  of  the  lesson  development  process 

3.  Author  aids  for  lesson  content  development 

4.  Author  aids  controlling  within  lesson  branching 

Lesson  "inquiry2"  (Test  Development) 

1.  Tutorial  author  aids  for  Test  Development 

2.  Author  aids  for  management  of  the  test  development  process 

3.  Author  aids  for  test  development  (e.g.,  test  instructions  and 
test  items) 

4.  Author  aids  for  post-test  reporting  of  results,  review  and 
remediation  actions. 


Lesson  Development  (Lesson  "inquiryl") 

1.  Tutorial  Author  Aids.  This  series  of  author  aids  provide  instruc¬ 
tional  system  designers  (authors)  with  guidance  in  the  preparation  of  lesson 
materials.  On-line  the  guidance  is  automatically  presented  at  appropriate 
points  during  the  lesson  development  process.  In  addition  to  the  forced 
presentation,  authors  may  review  any  specific  guideline  as  desired.  The 
guidelines  are  available  in  both  on-line  and  off-line  versions.  Specific 
guidelines  included  in  this  series  of  author  aids  are: 

•  Instructional  Sequencing  Rules 

•  Guidelines  for  the  Preparation  of  Terminal  Learning  Objectives 

•  Guidelines  for  Reducing  Reading  Difficulty  Level 

•  Guidelines  for  the  Preparation  of  Text  Material 

•  Guidelines  for  the  Use  of  Practice  Question  Formats  (General) 

•  Guidelines  for  the  Preparation  of  Multiple-Choice  Practice 
Questions 

•  Guidelines  for  the  Preparation  of  True-False  Practice  Questions 

•  Guidelines  for  the  Preparation  of  Constructed  Response 
Practice  Questions 

The  "off-line"  version  of  these  author  aids  will  be  of  use  for  all  instruc¬ 
tional  modes. 

2.  Author  Aids  for  the  Management  of  the  Lesson  Development  Process 

•  Sequencing  of  Instruction 

Instructional  content  is,  of  course,  based  on  Terminal  Learning 
Objectives  (TLOs) ,  Learning  Objectives  (LOs) ,  and  Learning  Steps  (LSs) . 
However,  how  these  are  sequenced  in  the  instruction  may  very  well  determine 
whether  an  instructional  module  is  effective  or  ineffective.  (A  module 
as  here  defined  begins  with  an  LO  or  LS  and  is  usually  followed  by  3-10 
frames  of  text  and  practice  questions  which  teach  the  LO  or  LS.)  To 
assist  authors  in  the  creation  of  modules  and  the  sequencing  of  instruction, 
worksheets  have  been  prepared  for  off-line  creation  of  LOs,  LSs,  text 


frames  and  various  types  of  practice  questions.  These  worksheets  can  be 
used  for  most  instructional  modes  but  are  particularly  suited  for  CAI  and 
programmed  instruction. 

•  Learning  Objectives  (LO)  and  Learning  Steps  (LS) 

Management 

Learning  Objectives  (LOs)  and  Learning  Steps  (LSs)  are  the  backbone  of 
the  IPISD  process.  They  dictate  the  content  of  both  instructional  material 
and  test  items.  The  instructional  system  designer  must  attend  to  them 
carefully  to  insure  they  are  represented  in  the  instruction  and  test 
situation.  As  described  above  (Sequencing  of  Instruction)  each  module  of 
instruction  begins  with  an  LO  or  LS.  The  author  is  required  to  input  the 
LO  or  LS  prior  to  inputting  instruction  for  a  given  module.  During  the 
preparation  of  the  instruction  for  a  module,  the  associated  LO  or  LS  is 
available  to  the  author  as  a  continual  reminder  of  the  instruction  to  be 
addressed.  A  by-product  in  the  CAI  version  that  is  available  to  students 
studying  a  particular  module  of  instruction  is  the  option  to  access  the 
LO  or  LS  statement  underlying  the  instructional  module  (see  Student 
Controlled  Branching  Author  Aid  below.)  The  "off-line"  version  of  this 
author  aid  will  be  of  use  for  all  instructional  modes. 

•  Reading  Difficulty  Index 

When  preparing  any  instructional  material  it  is  essential  for  the 
author  to  consider  the  intended  audience  for  the  material  [89-97],  There¬ 
fore,  an  author  aid  was  prepared  for  use  on  the  PLATO  system  that  auto¬ 
matically  computes  the  reading  difficulty  of  text  material,  question  stems, 
and  feedbacks  provided  the  student  as  the  material  is  inputted  into  the 
system. 1  In  using  the  aid,  authors  specify  the  reading  ability  level  of 
the  intended  audience  and  if  this  level  is  exceeded  the  computer  so  informs 
the  author  who  can  then  revise  the  material  to  a  lower  reading  level.  An 
off-line  version  provides  the  formula  and  identifies  the  components  required 
for  computing  the  reading  difficulty  index.  Obviously,  this  author  aid 
is  far  stronger  in  its  on-line  version  since  the  author  is  not  required  to 
compute  the  index.  However,  it  can  be  used  manually  in  off-line 
instructional  modes. 

3.  Author  Aids  for  Lesson  Content  Development 

•  Text  Creation  and  Editing 

This  author  aid  will  be  most  powerful  for  development  of  CAI 
materials.  In  CAI  form  it  permits  authors  to  create  CAI  executable  textual 
material  without  a  knowledge  of  the  programming  language  required  by  the 
system.  The  text  may  be  placed  at  the  author's  option  any  place  on  the 
screen  and  permits  revision  after  initial  creation.  It  incorporates  other 
author  aids  such  as  the  reading  difficulty  index  (described  earlier). 


^Available  in  Appendix  A. 
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This  is  actually  a  scries  of  several  author  aids  which 
creation  of  practice  questions.  The  anchor  aids  do  not  require 
by  authors  of  a  computer  programming  language.  The  aids  are  of 
value  in  a  CAI  or  PI  mode.  Detailed  characteristics  of  each  of 
author  aids  are  shown  in  Tables  1,  2  and  3. 


allows 
a  knowledge 
pr imary 
these 


4.  Author  Aids  Controlling  Within-Lesson  Branching 

0  Authov-Diveoted  Branching 

Today,  in  many  instructional  modes  such  as  CAI,  PI  and  other  forms 
of  self-paced  Instruction,  students  frequently  must  demonstrate  a  mastery 
of  current  instruction  before  being  allowed  to  go  on  to  new  instruction. 

Based  on  their  performance,  some  students  may  be  required  to  review  certain 
portions  of  the  instructional  material  while  others  will  go  through  the 
instruction  without  forced  review.  That  is,  students  are  branched  depend¬ 
ing  upon  their  particular  needs.  This  process  requires  that  student 
performance  be  continuously  monitored.  Author  aids  have  been  provided  to 
assist  authors  in  these  efforts.  Data  collection  aids  for  practice  questions 
provide  continuous  student  monitoring.  Other  aids  provide  guidance  to  the 
author  on  how  to  use -monitoring  information  (e.g.,  the  number  of  attempts 
a  student  is  permitted  at  a  practice  question,  the  conditions  under  which 
the  student  is  required  to  review  instruction,  etc.).  These  aids  have 
primary  application  in  self-paced  modes  of  instruction. 


0  Student-lH-ivated  Branching 


Students  themselves,  frequently  know  when  they  need  additional 
assistance  and  should  have  the  opportunity  of  accessing  this  assistance 
whenever  they  desire.  However,  they  must  be  able  to  identify  what  assistance 
is  available  and  the  means  for  accessing  it.  This  series  of  author  aids 
provides  students  the  options  of  accessing  auxiliary  information,  returning 
to  previously  studied  material^  and,  if  permitted  by  the  author,  of  branching 
to  the  end-of-lesson  test  from  anyplace  in  the  lesson.  These  aids  make  use 
of  aids  already  developed  for  other  purposes.  For  example,  the  management 
and  sequencing  aids  provide  specific  statements  of  TLOs,  LOs  and  LSs  asso¬ 
ciated  with  each  instructional  module.  The  student-directed  branching 
aids  permit  the  student  to  temporarily  branch  to  these  statements  whenever 
desired.  The  author  aids  provided  for  student-directed  branching  will  have 
wide  application  independent  of  the  instructional  mode  used. 


Test  Development  (Lesson  "inquiry2") 

1.  Tutorial  Author  Aids.  This  series  of  author  aids  is  similar  to 
those  discussed  in  the  tutorial  author  aids  for  lesson  development.  These 
aids,  however,  provide  both  on-line  and  off-line  guidance  in  the  various 


That  is. 


back-page,  return  to  beginning  of  lesson  or  beginning  of  module. 
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Table 


1 


CHARACTERISTICS  OF  PRACTICE  QUESTION  AUTHOR  AID 
(MULTIPLE  CHOICE) 


Question  stem  has  maximum  length  of  six  50-character  lines. 

Three-six  answer  alternatives  (including  correct  answer)  permitted. 

Each  answer  alternative  has  maximum  length  of  two  40-character 
lines. 

Correct  answer  position  randomly  selected.  (Author  may  select 
other  position  if  desired.) 

Author  "cued"  if  answer  alternatives  differ  in  length  by  more  than 
+  20  characters.  (Author  has  option  of  revising.) 

Author  specifies  one-three  attempts  student  permitted  on  question. 

Author  can  create  correct  answer  congratulatory  message.  Maximum 
of  five  40-character  lines. 

Incorrect  answer  feedback  messages  may  be  specific  to  response 
given,  or  general  feedbacks  which  may  be  different  for  different 
attempts.  Incorrect  answer  feedbacks  are  limited  to  five  40- 
character  lines. 

Correct  answer  given  student  if  number  of  permitted  attempts 
reached  without  student  correctly  answering  the  question. 

Reading  difficulty  of  question  stem  and  feedback  messages 
automatically  computed.  If  desired  reading  level  exceeded, 
author  has  the  option  to  revise  material. 


Table  2.  CHARACTERISTICS  OF  PRACTICE  QUESTION  AUTHOR  AID 
(TRUE-FALSE) 


•  Question  may  be  a  maximum  of  six  50-character  lines. 

•  Author  specifies  one  or  two  attempts  student  permitted  on  question. 

•  Author  can  create  correct  answer  congratulatory  message.  Maximum 
of  five  AO-character  lines. 

•  Incorrect  answer  feedback  message  provided  if  two  attempts 
permitted.  Messages  may  be  a  maximum  of  five  AO-character  lines. 

•  Correct  answer  given  student  if  number  of  permitted  attempts 
reached  without  student  correctly  answering  question. 

•  Reading  difficulty  of  question  and  feedback  messages  automatically 
computed.  If  desired  reading  level  exceeded,  author  has  the  option 
to  revise  material. 


Table  3.  CHARACTERISTICS  OF  PRACTICE  QUESTION  AUTHOR  AID 
(CONSTRUCTED  REPONSE) 


•  Question  may  be  a  maximum  of  six  50-character  lines. 

•  Author  specifies  one-three  attempts  student  permitted  on  question. 

•  Student  response  analyzed  for  one-four  correct  or  partially 
correct  answers. 

•  Student  response  analyzed  for  one-four  anticipated  incorrect 
answers . 

•  One-four  congratulatory  messages  permitted  depending  upon  number 
of  correct  or  partially  correct  answers  specified  by  author. 
Messages  may  be  a  maximum  of  five  40-character  lines. 

•  One-four  wrong  answer  messages  permitted  depending  upon  number 
of  incorrect  answers  specified.  Messages  may  be  a  maximum  of 
five  40-character  lines. 

•  Author  has  option  of  permitting  misspelling  of  answer;  words  in 
answer  to  be  out  of  order;  extra  words  in  answer;  and  disregarding 
the  capitalization  of  answer. 

•  Correct  answer  given  student  if  number  of  permitted  attempts 
readied  without  student  correctly  answering  the  question. 

•  Reading  difficulty  of  question  and  feedback  messages  automatically 
computed.  If  desired  reading  level  exceeded,  author  has  option 

to  revise  material. 


33 


facets  of  test  development.  Like  the  lesson  development  tutorial  author 
aids,  they  are  automatically  presented  to  authors  at  appropriate  points 
during  the  test  development  process  and  are  available  for  review  at  any 
time.  Specific  guidelines  included  in  this  series  of  author  aids  are. 

•  Guidelines  for  using  "inquiry2"  author  aids 

•  Guidelines  for  the  Preparation  of  Terminal  Learning 
Objectives 

•  Guidelines  for  Writing  Test  Instructions 

•  Guidelines  for  Reducing  Reading  Difficulty  Level 

•  Guidelines  for  the  Preparation  of  Multiple-Choice  Test 
Items 

•  Guidelines  for  the  Preparation  of  True-False  Test  Items 

•  Guidelines  for  the  Preparation  of  Constructed  Response 
Test  Items 

•  Guidelines  for  Assigning  Scores  to  Test  Items 

•  Guidelines  for  Post-Test  student  review  of  test  items 
and  Remediation  Strategies 

2.  Author  Aids  for  Management  of  the  Test  Development  Process 

•  Sequencing  of  Test  Items 

Worksheets  are  provided  to  authors  for  off-line  creation  of 
TLOs,  LOs,  LS  test  instructions,  and  test  items.  These  aids  permit  authors 
to  organize  and  sequence  their  test  items  prior  to  input  into  the  computer. 

The  worksheets  are  useful  for  all  modes  of  instructional  delivery. 

•  Terminal  Learning  Objectives  (TLO),  Learning  Objectives  ( LO ), 
and  Learning  Step  (LS)  Management 

This  author  aid  is  somewhat  different  from  the  corresponding  aid 
used  for  creating  TLOs,  etc.,  in  the  lesson  development  process.  Instruc¬ 
tional  system  designers  (authors)  input  all  TLOs,  etc.,  into  the  computer 
in  the  sequence  in  which  they  wish  to  cover  them  in  the  test.  (See  sequencing 
of  Test  Items  above.)  The  author  aid  then  maintains  records  of  which  TLO, 
etc.,  has  been  addressed  in  the  test  and  in  the  on-line  version.  The 
computer  "cues"  the  author  as  to  the  TLO  they  should  next  address.  The 
off-line  version  of  this  author  aid  will  be  of  use  for  all  instructional 
modes . 

•  Heading  Difficulty  Index 

This  author  aid  is  identical  to  the  one  discussed  earlier  in  the 
Author  Aids  for  Management  of  the  Lesson  Development  Process. 

3.  Author  Aids  for  Test  Development 

0  Creation  of  Test  Instructions  and  Editing 

This  author  aid  is  similar  to  the  text  creation  author  aid 
previously  described.  The  aid  will  be  most  useful  for  development  of 


student  Instructions  for  computer-administered  tests.  It  can  also  be  used 
for  on-line  development  of  instructions  for  other  instructional  modes  such 
as  programmed  texts,  etc.  The  use  of  the  author  aid  does  not  require  a 
knowledge  of  the  programming  language  required  by  the  system.  The  aid  also 
permits  revisions  to  be  made  to  the  instructions  after  initial  creation. 

0  Test  Item  Creation 

A  series  of  author  aids  was  developed  to  be  used  for  creation 
of  representative  types  of  multiple-choice,  constructed  response  and  true- 
false  test  items.  They  are  similar  to  the  practice  question  author  aids 
(see  Tables  1,  2  and  3)  except  that  they  do  not  provide  correct  and 
incorrect  response  feedbacks,  nor  a  variable  number  of  permitted  attempts. 

0  Timing  of  the  Test 

This  author  aid  permits  authors  to  establish,  if  desired,  a  time 
limit  for  individual  items  in  the  test  or  a  time  limit  for  the  entire  test. 

In  the  CAI  version  of  this  author  aid,  the  computer  maintains  a  record  of 
elapsed  time  and  takes  appropriate  action  based  on  the  elapsed  time.  This 
author  aid  is  most  useful  in  a  CAI  mode. 

0  Test  Item  Scoring 

Test  item  scoring  author  aids  are  provided  to  assist  test  developers 
in  establishing  test  scoring  procedures.  These  aids  include  such  consider¬ 
ations  as:  setting  cut-off  scores,  differential  weighting  of  various 
answers  to  a  test  item  (i.e.,  correct,  partially  correct,  and  incorrect 
answers),  and/or  differential  weighting  of  different  test  items.  Off-line 
versions  of  these  aids  consist  of  guides,  checklists,  etc.  On-line  versions 
are  similar  but  are  prepared  in  a  "query”  format.  The  aids  are  useful  for 
test  scoring  for  most  instructional  modes. 

4.  Author  Aids  for  Reporting  of  Results,  Review  and  Remediation 
Actions 

0  Reporting  of  Results 

Subsequent  to  test  item  scoring  (discussed  above)  authors  can 
establish  the  minimum  passing  score  required.  This  author  aid  then  scores 
the  test  and  automatically  reports  to  students  their  obtained  score  and 
the  minimum  score  required  for  passing.  The  aid  is  most  powerful  in  a 
CAI  format  but  may  be  also  used  in  other  self-paced  modes  of  instruction. 

0  Post-Test  Review  of  Test  Items  Missed 

These  author  aids  permit  the  test  developer  different  student 
review  options  for  test  items  missed.  For  example,  if  the  student  passes 
the  test  with  less  than  a  perfect  score,  the  author  may  elect  to  show  the 
student  the  correct  answer  to  items  missed.  Or,  in  the  case  of  students 
who  fall  the  test,  the  author  may  elect  to:  (1)  show  the  students  test 
items  missed  without  providing  the  correct  answers,  (2)  show  test  item'- 
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missed  and  include  the  correct  answers,  or  (3)  not  permit  a  review  of 
items  missed.  This  author  aid  is  primarily  of  use  in  a  CAI  or  programmed 
text  mode. 


0  Remediation  Aotions 

This  is  a  series  of  author  aids  which  permit  the  author  to  select 
the  type  of  action  that  will  be  taken  if  a  student  fails  the  test.  The 
actions  possible  in  these  aids  are  as  follows: 

(a)  Re-administration  of  instructional  lesson  followed  by.  re¬ 
administration  of  test  items  previously  missed. 

(b)  Re-administration  of  instructional  lesson  followed  by  re- 
administration  of  entire  test. 

(c)  Immediate  re-administration  of  test  items  previously  missed. 
(No  re-administration  of  instructional  lesson.) 

(d)  Immediate  re-administration  of  entire  test.  (No  re¬ 
administration  of  instructional  lesson.) 

(e)  Re-administration  of  test  items  missed.  Give  student 
option  of  reviewing  instructional  lesson  first. 

(f)  Re-administration  of  entire  test.  Give  student  option  of 
reviewing  instructional  lesson  first. 

(g)  No  re-administration  of  instructional  lesson  or  test — 
student  is  finished  with  lesson  or  goes  to  new  lesson. 

In  the  on-line  version  of  these  author  aids,  failing  students  are  auto¬ 
matically  branched  as  directed  by  the  author.  Therefore,  these  aids  are 
most  powerful  in  a  CAI  format.  However,  the  principles  underlying  the 
aids  can  be  employed  in  any  instructional  mode. 


TASK  3.  EVALUATION 


Three  levels  of  evaluation  were  undertaken  in  this  project.  The  first 
level  was  an  informal  evaluation  of  existing  IPISD  guidance,  procedures  and 
author  aids.  Six  HumRRO  personnel  with  technical  expertise  in  systems 
engineering  procedures  judged  the  ease  and  effectiveness  with  which  selected 
IPISD  procedures  and  guidelines  could  be  used  to  develop  Instruction.  Where 
appropriate,  these  author  aids  were  referenced  in  the  flowcharts  developed 
in  Task  1. 


The  second  level  of  evaluation  was  a  comprehensive  formative  evaluation 
of  the  new  author  aids  and  procedures  developed  for  application  to  IPISD 
Blocks  II. 2  and  III. 4.  Three  instructors  from  the  ENCOA  course  [one  officer 
and  two  NCOs  (E8)]  served  as  study  participants  and  as  a  data  source  for 
evaluation.  These  instructors  used  the  aids  to  create  test  items  and  lesson 
material.  Evaluation  data  were  gathered  as  the  authors  developed  their 
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instructional  material.  Formative  evaluation  of  the  author  aids  was 
accomplished  by  examining  author  performance  and  acceptance  of  the 
aids.  Critical  weaknesses  in  the  aids  (i.e.,  those  which  impeded  the 
progress  of  the  authors)  were  remedied  immediately  upon  diagnosis  of  the 
problems . 

In  the  third  evaluation  level,  the  adequacy  of  the  instruction  created 
by  the  military  authors  was  assessed.  This  instruction  was  administered  to 
U.S.  Army  Engineer  School  trainees  who  provided  the  data  for  evaluation. 

The  ultimate  criterion  of  instruction  is  evidence  of  desired  changes  in 
trainee  behavior  (i.e..  Does  it  "teach?").  In  order  for  the  author's  devel¬ 
opmental  activities  to  be  adequately  assessed,  trainee  performance  and 
attitude  data  were  collected. 

In  the  intial  stages  of  the  second  level  formative  evaluation,  HumRRO 
staff  functioned  as  "test  item  developers"  and/or  "preparers  of  lesson 
material."  Their  role  in  this  study  was  to  find  errors  and  faults  in  the 
directions,  requirements,  procedures,  etc.,  of  the  author  aids.  We  then 
used  these  data  to  make  needed  revisions  of  the  author  aids. 

Once  the  author  aids  were  considered  ready  for  application  to  actual 
course  content,  the  three  USAES  authors  were  given  training  in  using  the 
aids.  A  brief  15-minute  familiarization/training  period  in  using  PLATO 
preceded  each  individual's  involvement  in  the  project.  They  received 
instruction  in  signing  on  and  off  to  the  system  (which  included  signing 
into  the  appropriate  HumRRO  lesson).  A  brief  (approximately  5  minutes) 
orientation  to  the  PLATO  keyboard  was  then  presented  to  each  author.  This 
included:  use  of  the  edit  keys;  editing  techniques;  and  use  of  the  help 
sequence  keys  (e.g.,  HELP,  BACK,  NEXT,  etc.).  Descriptions  of  system 
crashes,  transmission  errors,  and  other  system  abnormalities  were  provided 
along  with  instructions  on  how  to  proceed  under  these  circumstances.  The 
authors  were  then  permitted  to  practice  with  the  keyboard  before  they 
started  inputting  their  lesson/text  materials,  and  all  of  them  chose  to  do 
so. 


Following  familiarization  training  on  PLATO,  the  three  USAES  instructors/ 
authors  were  given  a  brief  explanation  of  their  role  as  authors  and  then 
training  in  the  use  of  the  author  aids.  Project  staff  members  provided  the 
training  in  a  one-on-one,  tutorial  mode. 

Following  training  in  the  use  of  aids,  instructors  prepared  and  input 
on-line  in  the  PLATO  system  test  items  and  lesson  material.  Each  of  the 
instructors  developed  a  lesson  and  the  related  test  items  in  their  content 
specialty  as  part  of  a  2-3  hour  block  of  different,  but  related,  subject 
matter  from  the  Engineer  NCO  Advanced  (ENCOA)  course.  Table  4  lists  the 
subject-matter  blocks  selected  for  this  project. 

The  ENCOA  course  had  undergone  systems  engineering  and  USAES  personnel 
provided  a  set  of  well-defined  terminal  learning  objectives.  Test  items 
were  prepared  which  reflected  these  objectives.  An  additional  advantage  to 
the  ENCOA  course  was  that  both  NCO  and  Officer  instructors  were  available  as 
authors.  Hence,  the  utility  of  the  aids  could  be  evaluated  across  a  wide 
range  of  background  skills  and  experience. 
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Table  4.  ENCOA  LESSONS  AUTHORED  ON  PLATO  USING  INQUIRY  AIDS 


Lesson 

Author 

Time  to 
Create  Lesson 

Completion 
Time  in 

Current  Course 

Average 

Completion  Time 
in  CAI  Version 
(minutes) 

Field  Fortifications 

Emplacement 

Construction 

NCO-1 

48.5  hrs. 

4  hrs. 

94.3  (N  =  9) 

US/Foreign  Mine 
Warfare  Doctrine 

Officer 

41  hrs. 

2  hrs. 

60.2  (N  =  11) 

Protective  Mining 

NCO-2 

35  hrs 

3  hrs. 

38.6  (N  =  9) 

TOTAL 

124  hrs. 

9  hrs. 

191.1  minutes 
(or) 

3.22  hrs. 
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Although  each  author  was  required  to  Input  his  material  Into  the 
computer,  a  HumRRO  staff  member  was  present  to  assist  him  In  the  process 
on  a  one-to-one  basis.  No  Instructor  needed  to  know  the  TUTOR  language  or 
have  previous  TUTOR  experience  because  the  INQUIRY  author  aids  were  designed 
so  that  code  was  automatically  generated. 

Data  collection,  to  a  large  extent,  resulted  from  direct  observations 
of  the  authors  creating  and  inputting  their  instruction  and  from  structured 
Interviews  with  the  authors.  We  gathered  user  acceptance  data,  and  informa¬ 
tion  on  various  areas  of  difficulty  that  the  authors  experienced  while  using 
the  aids. 

The  authors  using  the  INQUIRY  system  were  encouraged  to  comment  at  any 
time  on  their  progress.  Monitors  were  present  at  every  inputting  session 
to  note  any  problems  encountered  or  comments  made  by  the  authors.  These 
comments  were  used  later  to  make  changos  in  the  system  so  that  it  was 
easier  to  use.  After  the  authors  inputted  all  of  their  material,  they 
received  a  questionnaire  asking  their  opinions  of  CAI  and  the  INQUIRY  system. 
In  addition  to  interview/questionnaire  data,  performance  data  were  collected. 
Such  items  as  the  time  to  create  a  given  frame  of  text  or  test  item  on-line, 
the  number  of  times  a  piece  of  text  had  to  be  re-input,  errors  in  attempts 
to  apply  a  particular  aid,  calls  for  help  from  the  monitor,  etc.,  were 
recorded. 

In  the  third  evaluation  level,  we  assessed  the  instruction  created  by 
the  authors  using  the  INQUIRY  aids.  To  the  extent  feasible,  student- 
identified  areas  of  difficulty  in  the  instruction  were  associated  with  the 
use  of  particular  author  aids.  In  this  way,  we  tried  to  determine  whether  a 
poorly  designed  aid  led  to  unclear  instruction  or  to  problems  with  the  tests. 

Twenty-two  students*  went  through  the  lesson  material  for  about  2-3 
hours  each  to  assess  the  quality  of  the  instruction  created  with  the  author 
aids.  All  students  received  preliminary  training  on  using  PLATO.  As  all 
the  students  could  not  go  through  all  the  instruction  and  testing  within 
the  time  allotted  by  USAES  for  this  project,  only  two  of  the  three  lessons 

were  presented  on  a  random  basis  to  each  student. 

Presentation  of  lesson  content  occurred  on-line,  as  did  the  administra¬ 

tion  of  the  post-tests  based  on  the  TLOs.  In  addition  to  collecting 
cognitive  data  regarding  student  performance,  exit  questionnaires  were 
administered  to  obtain  information  regarding  opinions  of  the  clarity  of 
the  instructional  material,  problems  encountered  in  the  practice  and  test 
items,  and  attitudes  toward  the  CAI  instructional  experience. 


Prerequisites  for  selecting  students  were  that  they  be  NCOs  who  have  entry 
qualifications  for  the  ENCOA  course,  but  have  not  been  exposed  to  the 
material  covered  in  these  lessons. 


39 


Findings 


7 .  Instruction  and  Test  Development.  The  tutorial  aids  were  presented 
to  each  author  prior  to  inputting.  In  no  instance  did  the  authors  seek  to 
reread  these  aids  which  presented  guidance  on  test  and  lesson  development. 
Thus,  it  cannot  be  concluded  whether  or  not  these  guidelines  were  useful  to 
the  authors.  It  appears  that  more  emphasis  on  the  applicability  and  value 
of  the  aids  is  required  in  order  for  authors  to  pay  attention  to  this 
guidance.  This  may  Involve  a  considerable  change  in  their  presentation 
format . 

As  a  result  of  the  initial  formative  evaluation,  the  authors  were 
able  to  prepare  test  and  lesson  material  with  minimal  difficulty.  In 
developing  almost  360  frames  of  instruction  and  testing,  a  total  of  65 
problems  were  experienced  by  the  authors  as  recorded  by  the  monitors.  Over 
25X  (18)  of  these  problems  were  trivial  errors  caused  by  the  author  pushing 
-.he  wrong  key.  Fourteen  instances  were  due  to  unclear  INQUIRY  instructions, 
which  were  remedied  as  soon  as  possible  after  they  were  noted.  Twenty-one 
problems  were  due  to  "bugs"  in  the  INQUIRY  program  which  were  eliminated  as 
soon  as  their  diagnosis  was  confirmed.  Six  problems  were  noted  as  due  to 
PLATO  system  crashes  and  transmission  errors.  Six  other  problems  arose 
from  miscellaneous  reasons.  Thus,  most  of  the  instructional  and  test  devel¬ 
opment  activity  undertaken  by  the  authors  occurred  smoothly  and  without 
undue  difficulty. 

2.  Time.  The  blocks  of  instruction  from  the  ENCOA  course  which  were 
put  on-line  are  traditionally  taught  in  9  hours.  The  average  completion 
time  was  under  3  1/4  hours  for  this  Instruction  including  taking  the 
associated  tests.  (See  Table  4) 


The  time  to  prepare  the  test  items  and  lesson  materials  using  the 
INQUIRY  system  of  author  aids  varied  little  from  one  author  to  another. 
Times  ranged  from  35  hours  for  one  NCO  (15  hours  on-line),  41  hours  for 
the  Officer  (14.5  hours  on-line),  to  48.5  for  the  remaining  NCO  (18.5  hours 
on-line) . 


3.  Readability  Index.  This  author  aid  provided  information  if  the 
reading  grade  level  was  surpassed  for  each  text  frame  or  test  item.  How¬ 
ever,  it  was  rarely  used.  That  is,  no  matter  what  the  index  showed,  the 
authors  chose  to  ignore  it.  About  220  text  frames  of  instruction  were 
produced  in  this  study.  More  then  50X  (126)  exceeded  the  pre-specif led 
reading  levels.  However,  only  1^  frame  of  instruction  was  revised  by  the 
author  as  a  result  of  this  information.  This  was  most  likely  due  either 
to  a  lack  of  confidence  in  the  measure's  validity,  or  to  a  lack  of  percep¬ 
tion  on  the  part  of  the  authors  regarding  the  criticality  of  reading  level, 
or  a  combination  of  both.  In  any  event,  no  changes  have  been  made  to  this 
aid  yet.  However,  we  believe  that  there  are  at  least  two  possible  changes 
needed.  First,  authors  should  be  given  more  instruction  in  the  usefulness 
of  this  aid  together  with  more  practice.  Second,  the  options  available  in 
the  INQUIRY  system  to  override  this  aid  should  be  removed  entirely  or  severely 
constrained  (i.e.,  within  1  grade  level  on  either  side  of  the  pre-specif ied 
one) . 
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4.  Multiple-Choice  Items.  As  authors  prepared  multiple-choice 
practice  and  test  items,  a  pre-programmed  INQUIRY  aid  assigned  the  correct 
answer  alternative  on  a  random  basis.  Authors  were  given  the  option  to 
change  the  designation  of  the  correct  answer  alternative,  and  approximately 
half  the  time  exercised  this  option.  Authors  were,  thus,  indicating  their 
preference  for  retaining  control  over  the  manner  in  which  they  created 
instruction. 

Another  author  aid  compared  the  lengths  of  answer  alternatives  and 
indicated  when  they  were  unequal.  This  occurred  in  about  half  the  items. 
However,  authors  unanimously  disregarded  this  information  and  left  the 
alternatives  as  they  were.  It  appears  that  more  restrictions  on  the  author 
aid  are  needed  in  order  for  these  author  aids  to  be  used. 

5.  Constructed  Response  Items.  In  the  constructed  response  format, 
authors  used  the  following  aids: 

•  The  aid  which  permitted  them  to  define  the  rigor  with  which 
answers  would  be  scored.  Authors  selected  those  options  which  permitted 
misspellings,  extra  words,  and  optional  capitalization.  However,  authors 
did  not  permit  the  words  in  the  answer  to  be  out  of  order. 

•  Authors  made  full  use  of  the  various  aids  available  for 
preparing  response  feedbacks  and  varied  between  providing  trainees 
specific  as  well  as  general  feedbacks  to  both  anticipated  and  unanticipated 
answers.  The  most  positive  reaction  by  students  was  to  the  explanatory 
feedbacks  presented  after  each  response  to  practice  questions.  The 
INQUIRY  author  aids  for  presenting  response  feedbacks  were  used  frequently 
by  the  authors  and,  if  possible,  should  be  incorporated  in  off-line 
instruction  (e.g.,  using  the  guidance  for  preparing  feedbacks  in  PI  texts). 

•  Authors  were  able  to  use  the  INQUIRY’  aids  to  specify  anticipated 
correct  and  incorrect  answers.  However,  there  appeared  to  be  a  problem 
with  anticipating  all  the  answers  which  were  given  by  the  trainees. 

The  student  attitude  questionnaire  data  indicated  a  strong  negative  reaction 
to  the  constructed  response  questions  provided  by  all  the  authors  both 
as  practice  and  as  test  items.  Student  performance  data  supported  this 
result,  as  most  difficulties  were  encountered  when  responding  to  constructed 
response  questions  (both  during  learning  and  test  taking).  These  results 
appear  to  be  due  to  those  instances  in  which  a  "correct"  answer  as  given  by 
the  student  is  considered  to  be  incorrect  by  the  system. 

The  monitors  had  observed  this  problem  as  authors  input  their 
material.  The  authors  could  not  adequately  anticipate  all  the  synonymous 
correct  answers  which  could  be  given  by  the  trainees.  This  problem  is 
particularly  critical  in  CAI,  as  the  evaluator  "knows"  whether  the  answer 
is  correct  after  seeing  it.  This  finding  suggests  that  the  guidance  for 
preparing  constructed  response  items  be  revised  to  clarify  situations  where 
authors  should  or  should  not  use  constructed  response  questions.  That  is, 
constructed  response  formats  should  be  used  only  in  cases  where  the  number 
of  possible  alternate  correct  answers  is  small. 
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6.  Student  Options.  Of  the  student  options,  the  ability  to  back  up 
to  a  previous  frame  (BACK)  was  considered  helpful  by  almost  all  of  the 
trainees  in  all  three  lessons.  The  other  three  options  were:  HELP — in 
which  the  relevant  learning  objective  was  displayed;  LAB — in  which  the 
student  could  return  to  the  beginning  of  the  lesson;  and  DATA — in  which 
the  student  could  go  back  to  the  beginning  of  the  module.  All  three 
options  were  rarely  used,  and  so  it  was  not  surprising  that  students  were 
divided  in  their  opinions  about  their  necessity. 


TASK  4.  REVISIONS 

Task  4  activities  consisted  of  making  revisions  to:  (1)  flowcharts 
developed  in  Task  1  and  (2)  author  aids  provided  in  Task  2.  The  purpose 
of  the  revisions  was  to  assure  maximum  utility  of  the  flowcharts  and  author 
aids  in  implementing  the  IPISD  process.  Revisions  constituted  a  series 
of  activities  which  spanned  almost  the  entire  research  period  and  paralleled 
all  development  and  evaluation  actions  in  the  other  Tasks.  Information 
sources  upon  which  revisions  were  based  are  as  follows: 

•  As  flowcharts  were  being  developed,  HumRRO  personnel  not  directly 
involved  in  the  project  provided  input  as  to  the  clarity,  utility  and  need 
for  revision. 

•  The  expanded  flowcharts  developed  in  Task  1  were  submitted  to 
instructional  system  designers  at  the  USAES  for  review. 

•  Review  of  the  Interim  Report  (which  contains  flowcharts)  by  the 
COTR  provided  additional  information  for  needed  revisions. 

•  As  on-line  author  aids  were  developed  they  were  initially  used  by 
HumRRO  personnel  to  identify  "bugs"  in  the  aids  which  were  corrected 
before  wider  use  was  made  of  them. 

•  The  most  important  test  of  the  utility  of  the  flowcharts  and  author 
aids  occurred  in  Task  3  when  authors  participating  in  the  research  effort 
used  the  flowcharts  and  author  aids  for  developing  instructional  material. 
Only  minor  "bugs"  were  identified  at  this  stage  since  the  flowcharts  and 
author  aids  had  undergone  extensive 'review  and  pre-testing.  Any  problems 
encountered  by  the  authors  in  using  the  flowcharts  and  author  aids  were 
immediately  corrected. 

•  The  last  information  source  for  flowchart  and  author  aid  revision 
was  to  occur  after  the  students  had  been  administered  the  instructional 
materials  developed  by  the  authors.  A  few  such  needed  revisions  were 
identified  as  a  result  of  difficulties  students  had  with  the  instruction 
that  was  directly  connected  with  the  author  aid  used  for  preparing  the 
instruction.  Specifically,  it  was  found  that  the  author  aids  for  preparing 
constructed  response  practice  and  test  items  require  additional  develop¬ 
mental  effort  in  guiding  authors  in  the  identification  of  what  constitutes 
a  correct  or  incorrect  answer.  For  example,  the  answer  to  a  question  might 
be  820  meters.  However,  if  the  student  answered  820  M  (which  should  be  an 
acceptable  answer),  they  were  judged  as  having  given  an  incorrect  response. 


As  a  result  of  the  Input  received  from  USAES  system  designers  and  the 
COTR,  major  formatting  revisions  were  made  to  the  flowcharts  contained  in 
the  Interim  Report  [88] .  The  revised  flowcharts  are  shown  in  Appendices 
B  and  C. 
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Chapter  V 

RECOMMENDATIONS 


The  evaluation  of  the  author  aids  reported  in  the  preceding  chapter 
has  demonstrated  the  feasibility  of  on-line  aids  for  implementing  IPISD 
Blocks  II. 2  (Develop  Tests)  and  III. 4  (Develop  Instruction).  User  accep¬ 
tance  of  the  aids  is  high  and  the  time  required  for  creation  of  test  and 
lesson  material  has  been  significantly  reduced.  Further  developmental 
effort  of  on-line  author  aids  appears  warranted.  Continued  development 
effort  should  include  five  major  areas  which  are  discussed  below.  These 
areas  are: 

1.  Modification  of  selected  current  author  aids  developed  in  the 
present  project. 

2.  Development  of  additional  author  aids  for  IPISD  Blocks  II. 2 
.  and  III. 4. 

3.  Development  of  author  aids  for  other  blocks  of  the  IPISD  model. 

4.  Conversion  of  aids  presently  programmed  for  PLATO  IV  CAI  to 
other  systems. 

5.  Author  Characteristics. 


MODIFICATION  OF  CURRENT  AUTHOR  AIDS 

There  was  insufficient  time  during  the  project  to  make  all  of  the 
modifications  that  were  indicated  during  formative  evaluation.  These 
modifications  should  be  made  if  the  lesson  and  test  development  author 
aids  are  to  be  maximally  effective.  The  specific  author  aids  for  which 
we  recommend  modification  are: 

(1 )  Reading  Difficulty  Index.  As  was  reported  in  the  previous 
chapter,  experimental  authors  did  not  revise  lesson  or  test  material  when 
the  material  was  written  at  a  reading  difficulty  level  in  excess  of  that 
intended.  Hence,  if  reading  level  is  critical,  the  author  aid  should  be 
modified  to  force  authors  to  revise  material  when  the  reading  level  is 
more  than  one  grade  above  that  desired. 

(2)  Author  Aid  for  Creating  Constructed  Response  Questions.  Authors 
require  additional  guidance  in  determining  how  to  use  constructed  response 
questions  appropriately.  When  constructed  response  questions  are  used, 
guidance  is  needed  in  the  selection  of  the  correct  answers  and  alternate 
forms  of  the  correct  ansyer  (e.g.,  George  Washington,  Geo.  Washington, 
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President  George  Washington,  etc.).  The  need  for  this  modification  arises 
from  the  level  3  evaluation.  Students  who  were  administered  the  experimental 
lessons  and  tests  had  difficulty  in  answering  constructed  response  questions. 
Often,  students  would  provide  answers  which  were  actually  correct  but  with 
an  answer  variation  not  anticipated  by  the  author.  Therefore,  their  answer 
would  be  erroneously  judged  incorrect.  When  this  happens,  it  is  very 
frustrating  for  the  student  and  if  it  occurs  frequently,  it  reduces  the 
perceived  instructional  value  of  the  lesson  to  the  student.  To  remedy  this 
situation,  authors  should  be  provided  with  detailed  guidance  on  the  use  of 
constructed  response  questions  as  well  as  guidance  on  the  framing  of  correct 
answer  variations. 

(3)  Editing  of  Test  and  Lesson  Material.  With  the  present  author  aids 
all  editing  must  occur  only  during  the  creation  of  text  or  questions.  Once 
material  has  been  completed  there  is  no  provision  for  further  editing. 

This  is  a  severe  weakness  of  the  present  author  aids.  It  is  possible  to 
revise  the  author  aids  so  as  to  permit  text  and  question  revision  after 
trial  administration  of  the  lesson.  However,  this  is  a  major  effort  out¬ 
side  the  scope  of  the  current  project. 

ADDITIONAL  AUTHOR  AIDS  FOR  IPISD  BLOCKS  II. 2  AND  Til. 4 

Although  additional  author  aids  could  have  been  developed,  this  was 
outside  the  current  scope  of  work.  Additional  aids  which  are  desirable 
include : 

•  Author  Aid  for  creation  of  Matching  Questions.  (This  aid  is 
presently  in  draft  form.) 

•  Author  Aid  for  creation  of  Arithmetic  Manipulation  Questions. 

•  Author  Aid  for  creation  of  Multiple  Choice  Questions  with  more 
than  one  correct  answer. 


CONVERSION  OF  INQUIRY  AUTHORS  AIDS  TO  A  CAI  SYSTEM  OTHER  THAN  PLATO  IV 

As  previously  stated,  the  PLATO  IV  CAl  system  was  considered  to  be  a 
research  vehicle  only.  The  goal  was  to  develop  and  document  on-line 
author  aids  that  could  be  programmed  on  any  CAI  system.  Therefore,  a  trial 
conversion  of  at  least  selected  author  aids  should  be  undertaken.  This 
undertaking  would  determine  the  extent  to  which  author  aids  developed  on 
one  CAI  system  could  be  converted  to  another  CAI  system  and  point  out 
difficulties  to  be  expected  in  such  a  conversion.  For  example,  rather  than 
using  the  TUTOR  language,  a  system-independent  language  such  as  PLANIT 
could  be  used  as  a  test  for  the  general  usefulness  of  the  on-line  author 
aids.  Use  of  flowcharts  which  supported  on-line  aid  development  on  PLATO 
IV  could  be  used  for  the  basis  for  this  effort. 


DEVELOPMENT  OF  AUTHOR  AIDS  FOR  IPISD  BLOCKS  OTHER  THAN  1 1. 2  AND  1 1 1. 4 

The  present  research  has  demonstrated,  to  a  degree,  the  utility  of 
on-line,  query-bused  author  aids  in  the  implementation  of  IPISD.  However, 
before  further  work  is  initiated  which  is  directed  toward  providing  on¬ 
line  author  aids  for  al.  1  applicable  Blocks  of  the  IPISD  mode,  other  research 
is  needed.  Examples  of  such  research  are  discussed  below.  Tills  study 
does  show  the  benefit  of  flowcharting  to  aid  the  IPISD  process,  and  such 
efforts  should  be  undertaken  for  other  IPISD  blocks. 

AUTHOR  CHARACTERISTICS 

Authoring  of  OAI  lessons  requires  a  certain  discipline  and  level  of 
competence  which  may  not  be  present  in  all  Instructors  assigned  to  this 
task.  Aids  are  thus  needed  which  constrain  the  author  much  more  than  was 
done  in  INQUIRY,  in  order  that  useful  guidance  and  techniques  can  be  applied 
In  creating  effective  instruction.  The  minimal  prerequisites  for  authoring 
both  on-  and  off-line  materials  need  to  be  established  as  well  as  the  extent 
to  which  aids  can  compensate  for  variable  experience  between  personnel. 

If  such  a  study  Indicates  that  many  Individual  proficiencies  are  lacking 
and  cannot  be  overcome  by  author  aids,  then  a  selection  and  classification 
problem  would  have  been  uncovered  and  an  assessment  of  "author"  Job/duty 
position  requirements  is  necessitated. 
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Appendix  A 

READABILITY  LEVEL  FORMULA 


Figure  the  average  length  of  a  sentence  in  number  of  words.  Figure  the 
average  word  length  in  number  of  letters. 

1.  Multiply  the  average  sentence  length  by  .5. 

2.  Multiply  the  average  word  length  by  4.71. 

3.  Add  the  products  of  Steps  1  and  2  together. 

4.  Subtract  21.43  from  the  sum  obtained  in  Step  3.  This  is  the 
readability  level  of  the  materials. 

Here  is  the  formula: 

[  (.5  (verage  sentence  length))  +  (4.71  (average  word  length))  ]  -  21.43 

(from  Kincaid,  1972) 
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Appendix  B 


FLOWCHARTS  OF  IPISD  BLOCK  1 1. 2  (DEVELOP  TESTS) 

AND  BLOCK  III. 4  (DEVELOP  INSTRUCTION) 

GUIDE  TO  THE  USE  OF  FLOWCHARTS 

The  IPISD  flowcharts  for  Blocks  II. 2  and  III. 4  (see  Figures  2  and  3 
in  Chapter  IV)  were  used  as  the  basic  framework  for  the  HumRRO-deve loped 
flowcharts.  The  flowcharts  in  this  appendix  expand  each  of  the  IPISD 
flowchart  blocks  into  detailed  step-by-step  components  which  must  be  per¬ 
formed  (or  considered)  in  completing  the  specific  flowchart  block.  (In 
the  appendix  IPISD  flowchart  blocks  are  noted  by  italics.) 

Flowchart  blocks  which  are  shaded  are  blocks  for  which  author  aids 
were  developed  in  this  project.  Next  to  these  shaded  blocks  are  indica¬ 
tors  specifying  whether  the  aid  is  on-line  and/or  off-line.  Blocks  with 
an  asterisk  (*)  next  to  them  indicate  that  existing  author  aids  have  been 
identified.  In  these  cases  we  provide  a  reference  to  the  author  aid  that 
is  to  be  used  at  that  specific  point  in  the  process  of  developing  tests 
and  instruction.  For  some  flowchart  blocks  supplementary  information  is 
presented  for  clarification  of  a  specific  block's  activity  statement. 

For  example,  Figure  B-l  shows  the  first  seven  task  elements  required 
to  prepare  multiple-choice  test  items.  The  total  task  elements  can  be 
found  on  pages  B-7  thru  B-9  of  this  Appendix. 

The  first  task  element  in  Figure  B-l,  Establish  Testing  Conditions 
for  Multiple-Choice  Tests  (2. 2. 1.2. 2)  is  the  task  to  be  performed.  The 
task  elements  under  this  block  must  be  performed,  or  at  least  considered 
when  preparing  multiple-choice  test  items.  For  example,  task  element  a, 

"set  readability  level  for  test,"  is  the  first  sub-task  shown  in  the 
figure.  The  asterisk  (*)  beside  the  block  indicates  the  availability  of 
a  non-HumRRO  author  aid.  In  this  case,  the  readability  level  set  is 
contingent  on  the  reading  level  of  trainees.  Since  this  block  is  shaded 
it  is  identified  as  a  block  for  which  an  author  aid  was  developed  in  this 
project.  This  aid  is  also  designated  as  both  an  on-line  and  off-line  author 
aid.  In  the  computer  version  of  this  author  aid,  the  author  is  specifically 
queried  as  to  the  reading  difficulty  desired  for  the  entire  test.  There¬ 
after,  all  material  input  by  the  author  is  automatically  checked  to  deter¬ 
mine  if  the  desired  readability  level  has  been  exceeded. 

Block  b,  "set  minimum  and  maximum  number  of  answer  alternatives 
including  the  correct  answer,"  indicates  that  author  aids  were  not  devel¬ 
oped.  Block  c,  "determine  if  more  than  one  answer  is  correct,"  has  neither 
an  asterisk,  nor  is  it  shaded.  This  indicates  that  no  off-line  author 
aid  has  been  identified  and  no  aid  was  developed  in  this  project  for  this 
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task  element.  The  comment  to  the  right  of  the  block  is  intended  to  further 
clarify  the  statement  within  the  block. 

Block  d,  "set  time  limits  if  any,"  is  shaded  identifying  it  as  requir¬ 
ing  development  of  an  author  aid  in  this  project.  In  the  computer  version 
of  this  aid,  authors  have  three  options:  (1)  an  untimed  test,  (2)  time 
limitation  for  individual  test  items,  and  (3)  time  limitation  for  entire 
test.  Again,  the  comment  to  the  right  of  the  block  is  for  further  clarifi¬ 
cation. 

Block  e,  "set  conditions  for  test  administration,"  has  neither  available 
or  developed  author  aids.  The  comment  further  clarifies  the  statement 
within  the  block. 

This  completes  the  task  elements  identified  as  required  for  establishing 
testing  conditions  for  multiple-choice  test  items.  The  next  major  task  to 
be  performed  is  the  actual  writing  of  the  multiple-choice  items  (2. 2. 1.2. 3). 
The  line  coming  out  of  the  block  indicates  that  in  the  actual  flowcharts 
tnis  task's  components  are  continued  on  subsequent  pages. 

The  flowcharts  in  this  appendix  are  on  the  pages  listed  below. 

Block  II. 2  (Develop  Tests)  —  Page  B-4 
Block  III. 4  (Develop  Instruction)  —  B-35 
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Figure  B-l.  Flowchart  for  Establishing  Test  Conditions  for  Multiple-Choice  Test  Items. 


* 

On  line 
and 

Off  line 


On  line 


On  line 
and 

Off  line 


*  Non-Hum  RRO  author  aid. 


Contingent  on  reading  level  of  trainees. 

See:  Peter  Kincaid,  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 

Author  aid  for  reducing  readability  level 


That  is,  is  one  (or  morel  answer  alternatives  correct,  in  addition 
to  correct  answer? 


Time  limit  for  each  item  and  entire  test. 


Will  all  trainees  take  the  same  test?  Will  they  all  be  in  the  same  room? 
Will  thev  be  tested  in  shifts?  (Some  of  these  decisions  are  contingent 
on  available  facilities  and  testing  personnel.) 


Author  aid  for  writing  multiple-choice  items. 

Author  aid  multiple-choice  initial  preparation  worksheet. 
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11.2  DEVELOP  TESTS 


if  learning 
category  is 
Mental  Skills 

? 

V  2. 2.0. 3  / 


Testing 

(Measuring) 

Attitudes 


Testing 

Information 


Testing 
Physical  Skills 


Consult  examples  below  to 
choose  learning  category  .1 


1  LEARNING  CATEGORIES 

Mental  Skills.  Skills  such  as 
Problem  solving 
Concept  formation 
Decision  making 


Mapreading 

Computer  programming 


Testing  Information-Knowing  standard  operating  procedures,  filling  out  clerical  forms 
Physical  Ski  I  Is -Perceptual  motor  skills,  such  as  typing,  target  shooting 
Testing  Attitudes-Cooperation,  dedication,  helpfulness,  leadership 


* 

On  line 
and 

Off  line 


On  line 


On  line 
and 

Off  line 


Contingent  on  reading  level  of  trainees. 

See:  Peter  Kincaid.  Automated  Readability  Index,  Human  Factois 
Society  Bulletin.  Vol.  XV,  No.  5,  May  1972. 

Authoi  aid  tor  reducing  readability  level. 


That  is,  is  one  (or  more)  answer  alternatives  correct,  in 
addition  to  correct  answer  7 


Time  limit  tor  each  item  and  entue  test 


Will  all  trainees  take  the  same  test?  Will  they  all  be  in  the  same  room? 
Will  they  be  tested  in  shifts?  (Some  of  these  decisions  are  contingent 
on  available  facilities  and  testing  personnel.! 


Author  aid  for  writing  multiple-choice  items. 

Author  aid  multiple-choice  initial  prepaiation  woiksheet. 


Non  HumRRO  authoi  aid. 
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2.2.1. 2.3 

_ I _ 


Make  sure  that  item  tests  one 
objective  only. 


a 


On  line 


* 

On  line 
and 

Off  line 


*Non-HumRRO  author  aid. 


Use  on-line  or  off-line  readability  aid.  Off-line  formula  is  at  end 
of  flowchart. 

Author  aid  on  reducing  readability  level. 

Peter  Kincaid,  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5.  May  1972. 


Errors  or  phrasing  of  question  might  inadvertently  indicate 
correct  answer. 


Negatives  are  confusing  to  test  takers.  State  items  positively. 


Do  not  make  distractors  overly  technical. 
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V.  ^ 

ii  nfri  him  ii 


Select  position  of  correct  answer 
alternative. 


Chosen  at  random  at  author's  option. 

Author  aid  question  scoring  form  for  recording  correct  answei 
position  and  number  of  alternatives. 


h 


Consult  subject  matter  experts 
or  peers  for  review  of  items. 


Consult  test  experts  for  final 
review  of  test. 


They  will  check  item  fidelity  and  correctness. 


They  will  point  out  possible  poor  construction  of  items  (and 
weak  or  ambiguous  items). 


* 

On  line 
and 

Off  line 


On  line 


On  line 
and 

Off  line 


*Non-HumRRO  author  aid. 


Contingent  on  reading  level  of  trainees. 

See:  Peter  Kincaid,  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 

Author  aid  for  reducing  readability  level. 


Will  the  items  be  50%  true  and  50%  false  or  ot*-er  proportions? 


Time  limit  for  each  item  or  entire  test. 


Will  all  trainees  take  the  same  test?  Will  they  all  be  in  the  same  room? 
Will  they  be  tested  in  shifts?  (Some  of  these  decisions  are  contingent 
on  available  facilities  and  testing  personnel.) 


Author  aid  for  writing  true/false  items. 

Author  aid  true/false  initial  preparation  worksheet. 
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* 

On  line 
and 

Off  line 


Make  sura  readability  level  is  not 
surpassed  in  item  stem. 


b 


*Non-HumRRO  author  aid. 


See:  Peter  Kincaid,  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 

Author  aid  for  reducing  readability  level. 


See  Stevens  and  O'Neil  (November  1974)  for  guidance 
and  examples. 


They  will  check  test  for  fidelity  and  correctness. 
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2.2. 1.2.5 


They  will  point  out  possible  bad  construction  of  items  (and 
look  for  weak  or  ambiguous  items). 


•Non-HumRRO  author  aid. 


Contingent  on  reading  level  of  trainees. 

Author  aid  for  reducing  readability  level. 

See:  Peter  Kincaid,  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 
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2.2.1.2.6 


Will  all  trainee*  take  test  in  same  room?  Will  all  trainees  take  same 
test?  Some  of  these  decisions  are  contingent  on  available  facilities 
and  testing  personnel. 


Author  aid  for  reducing  readability  level. 

See:  Peter  Kincaid.  Automated  Readability  Index.  Human  Factors 
Society  Bulletin,  Vol.  XV.  No.  5,  May  1972. 


Column  items  should  be  of  similar  category  (e.g.,  nouns,  verbs). 


*Non-HumRRO  author  aid. 
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2.2. 1.2. 7 


Label  each  column. 


Make  columns  with  unequal 
numbers  of  elements. 


This  is  so  answers  cannot  be  found  by  elimination. 


Consult  subject-matter  experts 
or  peers  for  review  of  tests. 


They  will  check  test  for  accuracy  and  fidelity. 


Consult  test  expert  for  final 
review  of  test. 


Ambiguous  items  or  those  of  poor  construction  will  be  found 
in  this  review. 


2.2.1.5 


* 

On  line 
and 

Off  line 


On  line 
and 

Off  line 


On  line 
and 

Off  line 


On  line 


Author  aid  for  reducing  readability  level. 

See:  Peter  Kincaid,  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5.  May  1972. 


Author  aid  for  preparation  of  constructed  response  items. 
Author  aid  of  examples  of  answer  alternatives. 


Author  aid  constructed  response  initial  preparation  worksheet. 


If  listing,  number  of  items  required?  What  part  of  speech 
is  missing? 


Time  limits  for  each  item  or  entire  test. 


Will  all  trainees  take  test  in  same  room?  At  same  time? 
Will  they  take  same  test?  Some  of  these  decisions  are 
contingent  on  facilities  and  available  testing  personnel. 


Non-HumRRO  author  aid. 
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Author  aid  (or  preparation  of  constructed  response  items. 


On  line 
and 

Off  line 


Make  sure  item  tests  one 
objective  only. 


a 


* 

• 

On  line 
and 

Off  line 


On  line 
and 

Off  line 


•Non-HumRRO  author  aid. 


List  possible  correct  answers. 
List  anticipated  incorrect 
answers. 

f 

- 1 - 


Make  sure  readability  level  of 
item  is  not  surpassed. 


b 


Write  completion/list  items  for 
Constructed  Response  Tests. 


2.2. 1.2.9 


Author  aid  for  reducing  readability  level. 

See:  Peter  Kincaid.  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 


If  fill-in-blank  format  used,  is  question  understandable 
with  blank? 


Author  aid  for  preparation  of  constructed  response  items. 
Author  aid  of  examples  of  answer  alternatives. 
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Consult  subject-matter  expert  or 
peers  for  review  of  test. 


They  will  look  for  fidelity  and  accuracy  and  may  add  other 
correct  answer  possibilities. 


Consult  text  expert  for  final 
review  of  test. 


This  review  will  pinpoint  poorly  constructed  items  or 
ambiguous  items. 


Establish  Testing  Conditions  for 
Constructed  Response  Tests 
Using  Essay  Items. 


Sat  readability  level  for  test 


Author  aid  for  reducing  readability  level 


See:  Peter  Kincaid.  Automated  Readability  Index,  Human  Factois 
Society  Bulletin.  Vol.  XV.  No.  5,  May  1972. 


Specify  number  of  alternatives  to  be  answered 


Determine  if  all  items  are  to  be 
used  or  if  a  choice  for  answering 
is  provided. 


Non-HumRRO  author  aid. 


2.2.1.2.10 


Will  all  trainees  take  the  same  test?  In  the  same  room?  In 
shifts?  Some  of  these  decisions  will  depend  on  available 
facilities  and  testing  personnel. 


Write  Essay  Type  Items  for 
Constructed  Response  Tests. 


2.2.1.2.11 


Make  sure  item  tests  one  objec¬ 
tive  only. 


2.2.1.2.11 


* 

On  line 
and 

Off  line 


Make  sure  readability  level  of  item 
is  not  surpassed. 


b 


Author  aid  for  reducing  readability  level. 

See:  Peter  Kincaid.  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 


Observe  time  it  takes  to  construct  answer. 


They  check  test  for  fidelity  and  accuracy. 


This  check  is  for  poorly  constructed  items  and  ambiguous  items. 


*Non-HumRRO  author  aid. 
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Develop  Oral  Test 


2.2.1.3 


Will  all  trainees  take  identical  tests?  How  will  facilities 
be  used? 


Author  aid  of  examples  of  answer  alternatives. 


Will  verbal  ability  be  part  of  answer  evaluation? 


Are  number  of  components  in  answer  to  be  delineated?  (For 
example,  explain  the  three  parts  of  an  experiment,  etc.) 
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2.2.1. 3 


On  line 
and 

Off  line 


On  line 
and 

Off  line 


On  line 


Author  aids  for  preparation  of  multiple-choice,  true/false  and 
constructed  response  items. 


Review  2. 2. 1.2.3,  .5,  .7,  .9,  and  .11  to  choose  best  format. 


Check  with  time  allotted  for  test. 


Author  aid  of  examples  of  answer  alternatives. 


They  check  for  fidelity  and  accuracy  and  may  add  other  correct 
answer  possibilities. 


*Non-HumRRO  author  aid. 


B-2 1 


Consult  test  expert  for  final 
review  of  test. 


This  will  highlight  ambiguous  and  poorly  constructed  items. 


Identify  facilities  to  be 
utilized. 


Find  out  if  any  restrictions  are  imposed  on  use  of  facilities 
and  personnel. 


a2 


2.2.1. 4 
_ I _ 


Set  up  schedule  for  use  of  specific  areas  and  schedule  for  personnel 
involved  in  testing. 


"County  Fair"  type  testing  environment  might  be  used. 


If  process,  the  behavior  will  have  sub  components 


Include  time  restrictions  and  facilities  to  be  used  in  descriptions. 
Describe  locations  and  duties  of  testing  personnel  in  description. 


See  Vineberg  and  Taylor,  1975. 
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2.2. t. 4 


This  will  be  necessary  for  scoring  later  on. 


They  will  check  for  fidelity  and  accuracy. 


This  check  will  illuminate  ambiguous  or  poorly  constructed 
exercises. 
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Author  aid  for  preparing 
test  instructions. 


Prepare  instructions  for 
administrator(s)  of  test. 


a 


Prepare  initial  and  within  test 
instructions  for  trainees 
taking  test. 


Check  readability  level  of  instructions  to  guarantee  that  reading 
level  has  not  been  surpassed. 

Author  aid  for  reducing  readability  level. 

Author  aid  for  developing  test  instructions. 


b 


S**  behavioral  performance  test  dev*. 

t  development  activities 


These  measures 
present  statements 
which  trainees  either 
"agree"  or  "disagree" 
with  (or  "like"  or 
"dislike). 


Perform  validity  check. 


Setting  of  criteria  is  sometimes  determined  by  reference 
to  job  requirements  or  consensus  of  "experts." 


Sometimes  trainees  are  required  to  “overlearn"  so  that  decay 
of  learning  on  the  job  is  not  detrimental  to  performance. 


Decide  on  number  of  LO's  to  be  met  by  trainee. 


Validity  shows  discrimination  between  masters  (those  who  are  "go‘ 
and  non  masters  (those  who  are  "no-go").  See  formula  at  end  of 
flowchart. 


a 


T 


2.3.1 


Decide  which  type  of  false  situation  is  more  critical  and 
which  can  be  tolerated. 


Develop  form  for  listing  purposes. 


Only  if  required. 


Swezey  says,  "If  the  cost  of  a  false  positive  (passing  an 
incompetent  man)  is  very  high,  the  cut-off  point  should  be 
set  very  high.”  (p.  6-13) 
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2.6 


Checklist  scoring  is  applicable  to  "process"  types  of 
performance. 


On  line 
and 

Off  line 


On  line 
and 

Off  line 
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2.6.2 


On  line 
and 

Off  line 


On  line 
and 

Offline 


*Non-HumRRO  author  aid. 


_ I _ 

Determine  whether  partially  cor¬ 
rect  responses  will  be  evaluated. 


d 


Determine  scheme  for  assessing 
partially  correct  answers. 


Author  aid  for  scoring  test  items. 

Author  aid  of  examples  of  answer  alternatives. 


Especially  applicable  in  constructed  response  (listing)  format. 
Author  aid  for  scoring  test  items. 

Author  aid  of  examples  of  answer  alternatives. 


Will  interference  scoring  be  used?  See  Swezey,  p.  6-6. 
Go/No-Go  type? 


Develop  form  for  recording  correct  and  incorrect  answers  of 
class  and  individuals.  Develop  method  for  determination  of 
number  of  correct  answers  (or  a  scoring  key). 


Author  aid  for  reducing  readability  level. 

See:  Peter  Kincaid,  Automated  Readability  Index,  Human  Factors 
Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 
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Identify  evei/eb/e  rttourv e*. 


Identify  personnel  knowledgeable 
in  verious  modes/medie  of 
instruction. 


Prepere  roster  of  coworkers 
(peers)  eveileble  for  instruction. 


Prepere  roster  of  eveileble. 
quetified  instructors. 


List  ell  librery/reference  looms,  study  rooms,  end  equipment 
clessroom  end  letge  lecture  hells;  (Meet ice  looms  'lebotetoiies. 
end  equipment 


Describe  end  document 
eveileble  fecilities. 


On  line 
and 

Off  line 


On  line 
and 

Off  line 


Base  decisions  on  criticality  of  task,  fidelity  required,  equipment 
availability,  personnel  availability. 

Will  learning  be  self  paced  (learner  controlled),  tutorial,  small 
group  or  large-group  discussion,  or  teacher  oriented^ 


Author  aid  for  preparing  learning  objectives. 
Author  aid  for  sequencing  of  mateiials. 


Author  aid  for  preparing  practice  frames. 


CHOOSE  FROM  LIST: 

2.3. 1  Audio  script. 

2.3.2  Video  materials 

2.3.3  Audiovisual  materials. 

2.3.4  Printed  materials. 

2.3.5  Programmed  instruction. 

2.3.6  Platform  lacturat. 


2.3.7  Salt  teaching  exportable 
packages  (STEPs). 

2.3.8  Supplementary  instruction. 

2.3.9  Adjunct  programs. 

2.3.10  Job  performance  aids  !JP As). 

2.3.11  Formal  OJT 

2.3. 12  Other  forms  of  mediated 
instruction. 

2.3.13  CAI 


B-38 


Review  TLO'sand  LO’s. 


See  Brown,  J.I.,  and  Cerlsen,  G.R.  Brown  Car Isen  Listening 
Comprehension  Test.  New  York:  Harcourt,  Brace  and 
World,  1955. 


*Non  HumRRO  author  aid. 
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Meet  with  producer  to  review 
•dipt 


»2 

_ _ _ 


Revise  script  if  necessary  to 
change  pace,  clarify  sections,  etc. 


g 


Prepare  required  number  of 
copies  of  script  tor  production 
purposes. 


h 


Make  recording  of  script. 


i 


Establish  sequence  of  visuals  to 
be  presented. 


Review  TLO’sand  LO's. 


Include  special  effects,  lettering,  shading,  amount  of  detail. 


c 


T 


*#18 


2.3.2 


Determine  whet  TV  facilities  are  required  and/or  available, 
(i.e.,  studio  equipment,  personnel,  materials,  etc.) 


Th»  audiovisual  production. 


Review  TLO's  and  LO’s. 


See  Brown,  J.t  ,  and  Catlsen,  G.R.  Brown-Catlsen  Listem 
Comprehension  Test  New  York:  Har court,  Brace  and 
World.  1955. 


•Non  HumRRO  author  aid. 


B 


Prepare  outline  of  instructional 
sections  applicable  to  audio¬ 
visual  medium. 


Left  side  should  contain  production  instructions  (use  of  music 
special  cues);  right  side  should  contain  directions  to  narrator. 


I- 


2.3.3. 1 

-  J _ 


Revite  script  if  necetiary  to  change 
pace,  clarify  aaefiont,  ate. 


9 


•Non-HumRRO  author  aid. 


See  Closed  Circuit  Television  Production  Technique!,  by  L.G.  Goodwin  & 
T.Koehring.  (Indianapolis:  Howard  W.  Santa  &  Co,,  (9701. 


Review  TLO’sand  LO's. 
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2.3.32 


Determine  what  photographic  facilities  required  and/or 
available  (i.e.,  studio,  equipment,  personnel,  materials,  etc.) 


See  The  Video  Handbook.  NY:  Media  Horizons,  1972. 


Refer  to  audiovisual  outlines. 


See  The  Video  Handbook.  New  York : 


Media  Horizons, 


*Non-HumRRC  author  aid. 
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Producing  a  television  program. 


Follow  steps  for  2.3.2  (Preparing  video  materials) . 

See  Closed  Circuit  Television  Production  Techniques. 


Indianapolis:  Howard  W.  Sams  and  Co..  1970. 


Developing  printed  materials 


Prepare  outline  of  instructional 
sections  applicable  to  print  medium. 


Review  TLO’sand  LO's. 


Determine  sequence  of 
instruction. 


Set  time  limits  for  using 
materials  in  class. 


Non-HumRRO  author  aid. 


Determine  location  and  type  of 
practical  exercises  and  illus¬ 
trations. 


b2 

Identify  reading 
level  of  trainees. 

comprehension 

c 

* 

On  line 
and 

Off  line 


frtahllati  raeH^illlti i  * - * 

troww  rMosoMiiy  ww  toc 


d 


Use  reading  test  scores. 


Author  aid  for  reducing  readability  level. 

See:  Peter  Kincaid,  Automated  Readability  Index,  Human  Factors 
Society  Bulletin.  Vol.  XV,  No.  5,  May  1972. 


Non-HumRRO  author  aid. 
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i 


Mwt  with  editofiii  ind  production 
staff  to  discuss  plans  for  instruc¬ 
tional  development. 


e4 


Sw:  Robert  Gunning.  How  to  Take  the  Fog  Out  of  Writing 
Chicago:  The  Oartnell  Corp..  1964. 


See  W.  James  Popham  and  Eva  Baker,  Planning  an  Instructional 
Sequence,  Englewood  Cliffs:  Prentice  Hall,  Inc.,  1970. 


Tyler  G.  Hicks.  Successful  Technical  Writing.  New  York: 
McGraw-Hill  Book,  Co.,  1959. 


Non-HumRRO  author  aid. 
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Have  printed  materials  reviewed. 


g 


Give  to  peer  for  review. 


9* 


Revise  if  necessary  to  clarify 
sections,  etc. 


h 


Have  sufficient  copies  prepared 
for  use  by  trainees. 


i 


* 


On  line 
and 

Off  line 


# 


Off  line 


•Non-HumRRO  author  aid. 


Review  TLO  standards  and  conditions.  Review  management 
plan.  See:  Thiagarajan  The  Programming  Process:  A 
Practical  Guide,  Worthington.  Ohio:  Charles  A.  Jones 
Publishing  Co..  1971. 


Author  aid  for  preparing  learning  objectives, 
Author  aid  for  sequencing  of  materials. 


Frames  on  the  main  track  (if  there  is  individualization)  should 
present  all  the  information  that  a  student  needs  in  order  to 
master  the  subject  matter. 


Relate  frame  size  to  expected  student  behavior.  Determine  how 
large  a  step  toward  mastery  the  student  can  reasonably  be 
expected  to  take  in  each  frame.  See:  Markle,  Good  Frames 
and  Bad,  New  York:  John  Wiley  &  Sons,  1964. 


Author  aid  for  preparing  practice  questions,  Author  aid  for 
instructional  frame  development. 
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-  e*  V 


2.3.5 

_ 1 _ 


Locate  areas  tor  practice  exercises. 


a4  1 


* 

On  line 
and 

Oft  line 


•Non  HumRRO  author  aid. 


Chooee  format  of  exerciaes. 


•4-2 


Determine  the  amount  of  practice  necessary  beyond  the 
minimal  range  of  examples.  See:  Markle,  1964. 


Multiple  choice,  true/false,  completion?  See:  Markle, 
Good  Frames  and  Bad,  New  York:  John  Wiley  &  Sons, 
1964. 

Author  aids  for  preparing  multiple  choice,  true/false 
and  constructed  response  practice  questions. 


Where  the  program  is  extensive  enough,  provide  isolated  teview 
and  test  items  as  feedback  to  the  programmer,  as  well  as  the  student 
on  how  well  the  teaching  sequence  has  gone.  Prepare  quir  which 
tests  students  understanding  of  the  material. 


Make  use  of  the  thematic  prompts  (in  context).  Only  use  formal 
prompts  (multiple-choice  format  excluded)  when  absolutely 
necessary.  See  Thiagarajan  &  Markle. 


It  the  student  is  branched  to  remedial  instruction,  the  material 
should  represent  a  restatement  of  infoimation  covered  in 
the  main  track. 


B-S7 


e*  -V 


Determine  it  there  are  supple 
mentary  materials. 


Provide  instructions  on  their  use. 


•Non  HumRRO  author  aid. 


To  be  used  with  supplementary  materials?  Alone? 


Will  feedback  contain  explanatory  material?  Will  examples 
be  used? 


See  Nesbit  and  O'neil,  Thiagarajan  &  Markle. 


Avoid  introducing  more  points  than  can  he  responded  to  in 
any  one  frame.  See  Markle. 


Provide  examples  covering  the  variety  of  conditions  the  student 
will  cope  with. 


B-'SB 


rv 


On  line 
and 

Off  line 


Check  readability  level  of  frames. 


See:  Peter  Kincaid,  Automated  Readability  Index,  Human 
Factors  Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 


Author  aid  for  reducing  readability  level. 


Off  line 


Eliminate  irrelevant  material 
from  frames. 


Write  practice  exercises  in 
rough  draft. 


♦  Choose  exercises  that  are  relevant 
to  instructional  content. 


See  Markle. 


Choose  the  number  of  exercises 
for  the  frames. 


Do  not  make  exercises  too  long  at  one  time 


Write  exercises  clearly,  in  good 
English. 


*Non-HumRRO  author  aid. 


Have  peer  review  rough  draft 
of  program. 


Try  out  draft  on  other 
authors. 


Have  editorial  review  of 
instructional  materials. 


See  Thiagarajan. 


Revise  materials  as  necessary. 


See  Nesbit  and  O'Neil,  Thiagarajan  and  Markle. 


Non-HumRRO  author  aid. 


Developing  platform  lectures. 


Review  TLO's  and  LO's. 


Refer  to  management  plan  for  time  constraints. 


Refer  to  management  plan. 


Note  different  concepts  to  get  across.  Also,  note  where 
supplementary  materials  are  useful 


B-hl 


2.3.6 

_L_ 


Determine  degree  of  student 
participation  in  lectures. 


Will  students  listen  only  or  will  interactions  be  encouraged? 


b 
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See:  Deterline  Associates,  How  to  Design  and  Develop  Self 
and  Supervised  Instruction:  A  Guide  for  Developing 
Correspondence  Instruction,  February  1975. 


See:  Preparing  Extension  Training,  TRADOC  Pamphlet  350-31. 
(Draft),  February  1976. 


*  Non-HumRRO  author  aid. 


Review  LO's  for  requirements. 


Identify  special  instructions  for  their  use,  if  any. 
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On  line 
and 

Off  line 


Developing  adjunct  instruction. 


Develop  outline  of  instruction 
to  be  covered  bv  adjunct 
instruction. 


Review  TLO's  and  LO’s. 


Identify  sequence  of  LO's  to 
be  covered. 


Identify  locations  in  sequence 
for  self-tests  and  exercises. 


Collect  and  review  all  materials 
to  be  used. 


Make  sun  that  text  dote  not 
excaed  reading  comprehension 
lave)  of  trainees. 


See:  Peter  Kincaid,  Automated  Readability  Index.  Human 
Factors  Society  Bulletin,  Vol.  XV,  No.  5.  May  \972. 

Author  aid  for  reducing  readability  level. 


Ensure  uniformity  of  length  and 
difficulty  of  instructional  unit. 


•  Non-HumRRO  author  aid. 


Writ*  procedural  instructions 
for  train**. 


d 


List  TLO's  and  LO's  to  be  attained 
bv  trainee. 


dl 


Write  correct  responses  for 
student  to  compare  his  answers 
on  quiz  or  test  items. 

d2 


Prepare  explanations  for 
possible  responses. 


d3 

:  i 

Prepare  branching  or  remedial 
instructions  based  on  responses. 

d4 

'  I 

Provide  suggestions  for  remedial 
exercises. 


d& 


Review  instruction. 


e 

r~ 


Develop  JPA's  f or  unit. 


# 


On  line 
and 

Off  line 


See.  Fully  Proceduralized  Job  Performance  Aids,  Handbook  foi  JPA 
Developers  by  Reid  P.  Joyce,  et.al.,  Air  Force  Human  Resources 
Lab,  AFHRL  TR  73-43(111). 


Include  references. 

Author  aid  for  preparing  test  instructions. 


Non-HumRRO  author  aid. 
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Have  peer  review  JPA  by  perform 
ing  tasks  using  JPA. 


Have  novice  perform  tasks,  noting 
any  problems,  ambiguities. 


Revise  JPA,  if  necessary 


Developing  form*/  OJT  (FOJTI 


Develop  outline  of  instructional 
sections  to  be  taught  by  FOJT. 


Review  TLO'sand  LO's 


2.3. 


_ _i 

Identify  which  sections  will  be 
presented  as  demonstrations, 
lectures,  "hands-on"  performance, 
or  mediated  instruction. 

b 

I 

Establish  sequence  of 
instruction. 


1 

Choose  locations  for  practice 
and  tests. 


cl 


Select  reference  materials 
and  aids. 


c2 


Prepare  FOJT  content. 


d 

I 

Obtain  assistance  from  subject 
matter  expert. 


dl 


Prepare  introductory 
material. 


d2 


Write  procedures  for  demon¬ 
strations  and  performance 
portions  of  instruction. 


See  blocks  2.3.3  and  2.3.6  for  further  reference. 


Prepare  instructions  for 
supervisor  to  score  student 
performance. 


f2 


2.3.11 


Author  aid  for  preparing  test  instructions. 
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Other  forms  of  mediated 
instruction. 


Identify  available  resources  and 
facilities  to  use  in  instructional 
development. 


Identify  available  personnel  who 
are  knowledgeable  in  using 
resources  to  prepare  mediated 
instruction. 


Prepare  instructional  sequence  an< 
strategies  depending  on  charac 
teristics  of  mediated  instruction. 


Establish  branching,  sequencing, 
and  remedial  paths  for  trainees. 


Prepare  text,  audiovisual  illus¬ 
trations,  exercises,  tests,  etc. 
that  are  suitable  for  the  medium 
to  be  used. 


ChMk  readability  of  textual 
materials. 


See:  Peter  Kincaid,  Automated  Readability  Index,  Human 
Factors  Society  Bulletin,  Vol.  XV,  No.  5,  May  1972. 


On  line 
and 

Off  line 


Author  aid  for  reducing  readability  level. 


•Non  HumRRO  author  aid. 
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See:  Handbook  for  Designers  of  Instructional  Systems, 
Air  Force  pamphlet  AFP-50-58,  Volume  IV,  July  1973. 


Author  aid  on  attitudinal  questionnaires  tor  CAI. 


See:  Handbook  ot  Procedures  tor  the  Design  ot  Instruction, 
Leslie  J.  Briggs,  Pittsburgh:  AIR,  1970. 


’Non-HumRRO  author  aid. 
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-  ,  / »  A 


Prepart  user  instructions. 


Guid«  for  the  use  of  author  aids  in  preparing  CAI  materials 


9 


Identify  degree  to  which 
instruction  treins  student  for  job. 


See:Handbook  for  Designers  of  Instructional  Systems  AFP  50  58 
July  1973.  '  - 


Non-HumRRO  author  aid. 
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Determine  duration  ot  training 
for  each  instructional  section. 


List  time  constraints  for  each  sequence,  practice  exercise,  etc 


Describe  instructor  require 
ments/duties  in  each  instructional 
section. 


Describe  media  resource 
requirements  related  to  training 
aids  and  facilities. 


List  the  available  facilities,  the  time  anti  duration  of  their  use  and 
the  supervisory  personnel  needed.  List  all  available  training  aids 
for  each  block  or  unit.  List  all  mediated  and  supplemental y 
instructional  materials,  their  location,  etc. 


Target  population  description. 


Identify  trainees'  academic  or 
educational  level,  reading  level 
verbal  ability,  etc. 


Refer  to  entry  tests. 


Identify  previous  training  or 
related  knowledge  and  experience 
of  trainees. 


List  courses  taken  and/or  hours  ot  training  in  specific 
prerequisite  areas. 


Identify  required  physical  and 
personal  characteristics  of  trainees 


List  required  physical  skills  and  chaiacteristics  (e.g.,  coordination, 
motor  skills)  ami  personal  qualities  (e  g.,  leadership,  motivation,  etc.) 


Identify  administrative  restraints. 


List  all  sequence  ranks  and  grades  of  students. 
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2.5 


List  schedule  for  using  facilities,  materials,  and  personnel 
resources. 


Discuss  needs  of  exceptionally  fast/slow  trainees. 


For  example,  ways  to  elicit  student  activity. 


For  example,  discuss  value  of  displaying  high  motivation 
of  teachers  and  easy  access  to  aids. 
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2.5 


For  example,  indicate  ways  to  change  pace  in  instruction. 


Describe  time  frame,  prerequisite  assumptions,  pre-tests,  it  any, 
post-test  requirements/criteria,  and  course  materials. 


Explain  sequence  of  course  lessons,  use  of  aids,  facilities,  and 
personnel  in  control  of  course,  remedial  branching,  if  present. 
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3.0  Outputs 


Describe  management  of  course  and  students. 
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Appendic  C 


DETAILED  FLOWCHARTS  OF  AUTHOR  AIDS  FOR 
IPISD  BLOCK  II. 2  (DEVELOP  TESTS)  AND  BLOCK  III. 4  (DEVELOP  INSTRUCTION) 


The  two  sections  of  flowcharts  in  this  appendix  serve  three  purposes. 

First,  they  further  clarify  IPISD  Blocks  II. 2  and  III. 4.  The  number  at  the 
beginning  of  each  flowchart  refers  to  the  appropriate  place  they  are  to  be 
inserted  in  the  IPISD  blocks  flowcharted  in  Appendix  B.  Additional  block 
identification  was  not  made  because  of  the  complexity  of  the  flowcharts. 

The  second  purpose  of  the  flowcharts  is  to  provide  directions  for  the 
use  of  the  on-line  authoring  aids  for  test  and  lesson  development.  These 
flowcharts  are  detailed  enough  to  lead  an  author  from  the  beginning  of 
development  (the  learning  objective)  to  the  finished  product  (on-line 
for  student  use) . 

\ 

The  third  purpose  of  the  flowcharts  is  to  show  the  logical  flow  of  the 
author  aids  (including  where  they  fit  into  the  larger  IPISD  process),  so 
that  conversion  of  the  aid  is  possible  on  other  computer-administered 
instructional  systems.  On-line  demonstrations  of  the  flowcharts  are  possible 
by  accessing  HumRRO  lessons  inquiryl  (test  development)  and  inquiry2  (lesson 
development  on  the  PLATO  system. 

The  flowcharts  in  this  appendix  are  on  the  pages  listed  below. 

Preparing  CAT  Test  Materials  —  Page  C-2 
Preparing  CAT  Lesson  Materials  —  Page  C-30 
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II. 2.2.1. 1.b 


PREPARING  CAI  TEST  MATERIALS 
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I 


Multiple  choice  item  format. 


2 


"  “V".  ^  ''  ' a, 

-  r  -«-»•  •  -**•  V,  * 

•  ►  ****■■  v  c* ^ 

—•  •  • - ,1  ‘ 


"  •“  i  iilSfitiitt  --1 _ . 


j 


/  Have  \ 
/  guidelines  \ 
Qor  entering  answers 
\been  studied^/ 


Study  examples. 


'  Assistance 
needed  to  enter 

\  answers 

?  / 


Enter  as  many  as 
6  correct,  partially 
correct  or  incorrect 


Ignore 

capitalization 

? 


Type  changes  if 
necessary. 


Complete  scoring 
form  for  item. 


C—  1 7 
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D 


Stop  temporarily. 


C-19 


4  )  Create  initial  test  instructions 


C-23 


C-24 


ministration 
of  instructional 


/ readministra\ 
tion  of  test 
items  previously 


Type  in 
option  1 


readministra¬ 
tion  of  entire 


Type  in 
option  2 

1 

- 1 
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III. 4.2.3.13 


PREPARING  CAI  LESSON  MATERIALS 


to 


1 


Type  in  data  set  name 
and  lesson  name. 
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Text  creation 


C-34 


a*  X 


C-35 


Practice  frame  creation. 
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1  Z*'l  **V 
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N 


N 


C-4  3 


/  Anticipated 
wrong  answers 
.  for  item 


Study  examples. 


/  Assistance  \ 
'needed  to  enter 
anticipated  wrong 


Enter  as  many  as 
four  anticipated 
wrong  answers. 


Make  revisions. 


mmm 


Type  in  number 
of  attempts 
permitted. 


Revise  number 
of  attempts. 


Are 

attempts 
greater  than  2 


f  Different  ' 
try  again 
feedbacks  for 
each  attempt 


Are 

3  attempts 
permitted 


Type  general 
try  again  feedback. 

./lading  level  \Y  /  f 

Make  revisions. 

■  1  v  ■ ..  /  A  feedback  / 

\  exceeded  /  \  ?  / 

V/  \  / 
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Use  learning 
objective 
prepaiation 
worksheet. 


/  Learning  \ 
objective  created 
off  line 


Enter  learning 
objective. 


Revise 

objective 


reading  level 
exceeded 


Return  to  index 


guideline 


Type  in  guideline 
number. 


Review  guidelines 


Another 

guideline 


N 


Stop  temputanly . 
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Stop,  lesson  is  finished 


Type  in  new 
lesson  name  for 
final  storage  of 
instructions. 


Lesson 

for  formatting 
instruction 
\  changed  > 


See  manual  for 

conversion 

instructions. 


Review  lesson. 


End  CAI  lesson 
creation. 


See  Appendix  B 
page  B-79. 


Best 

Available 

Copy 
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1  Military  Attache.  French  Embassy  ATTN  Doc  Ser 
1  Medecm  Chef,  C.E  R  P  A  Aisenal,  Toulon.  NavJ  T  iant 
1  Pun  Scientific  Oft.  Appl  Hum  Engr  Rsch  Div,  Mmistiy 
ol  Defense.  New  Delhi 

1  Pers  Rsch  Otc  Lilxaty.  AKA,  Israel  Defense  Fon.cs 
1  Minister  is  van  Defensie,  OOOP/KL  Afd  Sociaaf 
Psycbologische  Zaken,  The  Hague  Netherlands 


